Jul 23 21:51:58.678415 ip-10-0-142-110 systemd[1]: Starting Kubernetes Kubelet... Jul 23 21:52:01.514093 ip-10-0-142-110 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 21:52:01.514093 ip-10-0-142-110 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 21:52:01.514093 ip-10-0-142-110 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 21:52:01.514609 ip-10-0-142-110 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 21:52:01.514609 ip-10-0-142-110 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 21:52:01.516673 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.516587 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 21:52:01.524415 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524395 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.524415 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524413 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.524415 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524417 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524420 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524423 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524426 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524429 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524432 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524435 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524447 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524450 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524452 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524455 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524458 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524460 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524463 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524466 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524470 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524473 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524475 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524491 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524495 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.524540 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524497 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524500 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524503 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524506 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524509 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524511 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524514 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524516 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524519 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524521 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524523 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524526 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524528 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524532 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524535 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524538 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524540 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524543 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524546 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524548 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.525165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524557 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524560 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524563 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524565 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524567 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524570 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524572 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524575 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524578 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524580 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524582 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524585 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524587 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524589 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524592 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524594 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524596 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524598 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524601 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524609 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.525693 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524612 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524614 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524617 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524619 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524622 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524625 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524628 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524630 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524633 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524635 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524638 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524642 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524645 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524658 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524661 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524663 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524669 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524672 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524675 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.526128 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524678 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524681 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524684 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524687 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524689 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524692 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524694 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524697 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524699 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524703 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524707 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524710 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524713 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524716 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524718 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524721 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524724 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.524727 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526189 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.526565 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526199 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526203 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526208 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526211 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526214 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526216 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526219 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526221 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526224 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526226 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526229 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526231 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526234 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526236 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526239 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526243 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526245 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526248 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526250 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526253 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.527025 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526255 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526258 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526260 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526262 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526265 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526267 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526270 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526272 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526275 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526278 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526280 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526282 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526286 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526289 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526291 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526294 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526296 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526298 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526301 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526305 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.527500 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526308 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526318 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526321 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526324 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526326 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526329 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526332 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526336 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526339 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526342 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526344 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526347 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526349 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526352 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526354 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526357 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526359 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526362 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526364 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526367 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.528011 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526370 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526372 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526374 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526377 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526379 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526382 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526384 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526387 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526389 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526392 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526395 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526397 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526400 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526402 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526405 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526408 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526411 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526413 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526415 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526418 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.528458 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526420 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526423 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526425 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526427 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526430 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526432 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526434 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526437 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526440 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526442 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526444 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526447 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526449 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526452 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526454 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526456 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526459 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.526462 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527216 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527222 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.528918 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527225 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527228 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527231 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527234 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527236 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527239 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527242 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527244 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527247 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527249 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527251 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527254 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527256 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527258 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527261 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527265 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527269 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527272 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527275 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.529365 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527277 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527280 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527283 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527285 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527288 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527290 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527293 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527295 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527298 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527300 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527303 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527305 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527308 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527310 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527313 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527316 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527318 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527321 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527323 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527326 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.529811 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527329 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527331 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527334 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527336 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527339 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527341 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527344 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527346 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527348 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527351 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527353 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527355 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527358 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527360 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527362 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527365 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527367 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527370 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527373 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527375 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.530273 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527378 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527380 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527382 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527385 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527388 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527391 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527394 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527396 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527399 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527401 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527403 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527406 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527408 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527411 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527415 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527417 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527420 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527423 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527425 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.530732 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527429 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527432 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527435 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527438 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527441 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527443 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527446 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527449 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527452 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527454 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527456 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527459 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527461 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527464 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527466 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527468 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527471 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527473 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.527476 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527568 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527575 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 21:52:01.531187 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527581 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527585 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527589 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527593 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527597 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527602 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527605 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527609 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527612 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527615 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527618 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527621 2569 flags.go:64] FLAG: --cgroup-root="" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527625 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527627 2569 flags.go:64] FLAG: --client-ca-file="" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527630 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527633 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527637 2569 flags.go:64] FLAG: --cluster-domain="" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527640 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527643 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527646 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527650 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527653 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527656 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 21:52:01.531715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527659 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527662 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527665 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527668 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527671 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527674 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527676 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527682 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527685 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527687 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527690 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527693 2569 flags.go:64] FLAG: --enable-server="true" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527696 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527700 2569 flags.go:64] FLAG: --event-burst="100" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527703 2569 flags.go:64] FLAG: --event-qps="50" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527706 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527709 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527712 2569 flags.go:64] FLAG: --eviction-hard="" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527720 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527723 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527725 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527728 2569 flags.go:64] FLAG: --eviction-soft="" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527731 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527733 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527736 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 21:52:01.532271 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527739 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527742 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527744 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527747 2569 flags.go:64] FLAG: --feature-gates="" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527751 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527754 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527757 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527760 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527762 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527765 2569 flags.go:64] FLAG: --help="false" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527768 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527771 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527774 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527776 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527781 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527784 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527787 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527789 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527792 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527795 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527797 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527800 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527803 2569 flags.go:64] FLAG: --kube-reserved="" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527806 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 21:52:01.532869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527809 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527812 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527815 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527817 2569 flags.go:64] FLAG: --lock-file="" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527820 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527823 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527826 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527831 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527834 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527836 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527839 2569 flags.go:64] FLAG: --logging-format="text" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527842 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527846 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527848 2569 flags.go:64] FLAG: --manifest-url="" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527851 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527855 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527858 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527862 2569 flags.go:64] FLAG: --max-pods="110" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527865 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527869 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527871 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527874 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527878 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527881 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527884 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 21:52:01.533421 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527891 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527894 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527897 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527900 2569 flags.go:64] FLAG: --pod-cidr="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527903 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527907 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527910 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527913 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527916 2569 flags.go:64] FLAG: --port="10250" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527919 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527922 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e81ed2fbd837e645" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527925 2569 flags.go:64] FLAG: --qos-reserved="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527928 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527931 2569 flags.go:64] FLAG: --register-node="true" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527934 2569 flags.go:64] FLAG: --register-with-taints="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527937 2569 flags.go:64] FLAG: --registry-burst="10" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527940 2569 flags.go:64] FLAG: --registry-qps="5" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527942 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527945 2569 flags.go:64] FLAG: --reserved-memory="" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527951 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527954 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527957 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527959 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527962 2569 flags.go:64] FLAG: --runonce="false" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527965 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 21:52:01.534005 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527968 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527971 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527973 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527976 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527980 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527983 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527986 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527989 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527992 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527995 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.527998 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528001 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528004 2569 flags.go:64] FLAG: --system-cgroups="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528006 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528011 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528014 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528018 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528021 2569 flags.go:64] FLAG: --tls-min-version="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528024 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528027 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528029 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528032 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528035 2569 flags.go:64] FLAG: --v="2" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528039 2569 flags.go:64] FLAG: --version="false" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528043 2569 flags.go:64] FLAG: --vmodule="" Jul 23 21:52:01.534790 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528046 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.528051 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528158 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528162 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528164 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528167 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528169 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528173 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528175 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528178 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528180 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528184 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528186 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528189 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528191 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528194 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528196 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528199 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528202 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528206 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.536292 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528208 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528211 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528213 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528216 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528219 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528221 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528223 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528226 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528228 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528230 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528233 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528235 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528237 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528241 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528244 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528246 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528249 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528251 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528254 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.536778 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528256 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528258 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528261 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528263 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528267 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528270 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528272 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528274 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528277 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528280 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528282 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528285 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528287 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528290 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528292 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528295 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528297 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528299 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528302 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.537218 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528304 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528308 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528312 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528314 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528317 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528320 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528322 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528327 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528329 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528332 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528335 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528337 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528339 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528342 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528344 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528347 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528349 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528353 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528355 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528358 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.537677 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528360 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528362 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528365 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528367 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528370 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528373 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528377 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528379 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528382 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528385 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528387 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528390 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528393 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528395 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528398 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528400 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528403 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528405 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528408 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.538129 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528412 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528414 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528416 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.528419 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.529121 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537225 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537241 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537260 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537266 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537320 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537325 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537329 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537332 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537335 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537338 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.538589 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537341 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537344 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537347 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537350 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537353 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537356 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537359 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537362 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537364 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537367 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537369 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537372 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537374 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537377 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537379 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537382 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537384 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537387 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537389 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537391 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.538952 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537394 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537397 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537400 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537403 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537407 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537411 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537414 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537416 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537419 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537422 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537424 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537427 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537430 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537432 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537435 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537437 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537439 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537442 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537445 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537447 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.539412 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537451 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537454 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537456 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537459 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537461 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537463 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537466 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537468 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537471 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537473 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537476 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537497 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537500 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537504 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537508 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537511 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537514 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537517 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537520 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537522 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.539900 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537525 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537528 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537530 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537533 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537535 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537538 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537540 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537543 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537545 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537548 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537550 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537553 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537556 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537559 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537563 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537566 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537569 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537571 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537574 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.540392 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537577 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537579 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537582 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537584 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537586 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537589 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537592 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537594 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537597 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537599 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537602 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537604 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537607 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537609 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537615 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 21:52:01.540850 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537723 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537728 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537731 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537733 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537736 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537739 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537742 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537744 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537747 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537749 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537752 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537754 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537757 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537760 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537762 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537765 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537767 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537770 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537772 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537774 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 21:52:01.541226 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537777 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537780 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537782 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537784 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537787 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537790 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537793 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537795 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537798 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537800 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537804 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537808 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537811 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537814 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537817 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537819 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537822 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537824 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537827 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537829 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 21:52:01.541713 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537832 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537834 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537837 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537839 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537841 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537844 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537847 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537849 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537852 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537854 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537857 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537859 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537861 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537864 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537866 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537869 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537871 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537874 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537876 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537880 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 21:52:01.542185 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537883 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537886 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537889 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537891 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537894 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537897 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537901 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537903 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537905 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537908 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537910 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537912 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537915 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537917 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537920 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537922 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537925 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537927 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537930 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 21:52:01.542659 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537933 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537935 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537937 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537940 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537942 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537945 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537947 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537950 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537952 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537954 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537957 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537959 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537962 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537964 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537966 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537969 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537971 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537974 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537976 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 21:52:01.543108 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:01.537979 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.537983 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.538108 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.540084 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.541495 2569 server.go:1019] "Starting client certificate rotation" Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.541594 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 21:52:01.543560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.541624 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 21:52:01.543766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.543754 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 21:52:01.548510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.548448 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 21:52:01.557755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.557740 2569 log.go:25] "Validated CRI v1 runtime API" Jul 23 21:52:01.561455 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.561437 2569 log.go:25] "Validated CRI v1 image API" Jul 23 21:52:01.562281 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.562264 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 21:52:01.565684 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.565661 2569 fs.go:135] Filesystem UUIDs: map[042d0083-e9f6-43df-a5af-6144d693a0b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e57b2b9a-af0d-4f85-af1b-6b415e16eba8:/dev/nvme0n1p4] Jul 23 21:52:01.565739 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.565684 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 21:52:01.572088 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.571964 2569 manager.go:216] Machine: {Timestamp:2026-07-23 21:52:01.570413749 +0000 UTC m=+1.595747275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200289 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b059d6a06ec50e30be7a9d5f33882 SystemUUID:ec2b059d-6a06-ec50-e30b-e7a9d5f33882 BootID:ed1d7518-8a52-44a7-b18f-9c00f7225a3f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:04:17:db:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:04:17:db:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:83:16:55:d5:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 21:52:01.572088 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.572076 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 21:52:01.572214 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.572187 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 21:52:01.575319 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.575290 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 21:52:01.575448 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.575320 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 21:52:01.575925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.575916 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 21:52:01.575957 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.575928 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 21:52:01.575957 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.575935 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 21:52:01.576546 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.576536 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 21:52:01.577453 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.577439 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 21:52:01.577690 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.577680 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 21:52:01.580579 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580567 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 23 21:52:01.580694 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580684 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 21:52:01.580732 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580713 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 23 21:52:01.580732 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580723 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 21:52:01.580785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580739 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 21:52:01.580785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580748 2569 kubelet.go:398] "Adding apiserver pod source" Jul 23 21:52:01.580785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.580756 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 21:52:01.581071 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.581060 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 21:52:01.581108 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.581077 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 21:52:01.583234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.583219 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 21:52:01.585208 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.585191 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 21:52:01.585291 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.585229 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 21:52:01.586353 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586339 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586359 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586369 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586378 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586391 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586400 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586412 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 21:52:01.586426 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586419 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 21:52:01.586654 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586440 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 21:52:01.586654 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586450 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 21:52:01.586654 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586468 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 21:52:01.586654 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.586493 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 21:52:01.587356 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.587344 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 21:52:01.587413 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.587359 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 21:52:01.590562 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.590532 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 21:52:01.590650 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.590596 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 21:52:01.590650 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.590630 2569 server.go:1268] "Started kubelet" Jul 23 21:52:01.590757 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.590646 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 21:52:01.590806 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.590750 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 21:52:01.590849 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.590817 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 21:52:01.591372 ip-10-0-142-110 systemd[1]: Started Kubernetes Kubelet. Jul 23 21:52:01.591830 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.591457 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 21:52:01.592397 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.592360 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 21:52:01.593819 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.593805 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 23 21:52:01.596119 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.596102 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 21:52:01.596834 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.595950 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-110.ec2.internal.18c509b9f410c9e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-110.ec2.internal,UID:ip-10-0-142-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-110.ec2.internal,},FirstTimestamp:2026-07-23 21:52:01.590610401 +0000 UTC m=+1.615943929,LastTimestamp:2026-07-23 21:52:01.590610401 +0000 UTC m=+1.615943929,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-110.ec2.internal,}" Jul 23 21:52:01.599432 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.599415 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 21:52:01.599432 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.599424 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 21:52:01.599617 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.599521 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:01.600064 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600002 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 21:52:01.600064 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600061 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 21:52:01.600177 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600076 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 21:52:01.600177 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600172 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 23 21:52:01.600262 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600179 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 23 21:52:01.600262 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600259 2569 factory.go:55] Registering systemd factory Jul 23 21:52:01.600352 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600292 2569 factory.go:223] Registration of the systemd container factory successfully Jul 23 21:52:01.600551 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600536 2569 factory.go:153] Registering CRI-O factory Jul 23 21:52:01.600551 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600550 2569 factory.go:223] Registration of the crio container factory successfully Jul 23 21:52:01.600676 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600592 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 21:52:01.600676 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600610 2569 factory.go:103] Registering Raw factory Jul 23 21:52:01.600676 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600620 2569 manager.go:1196] Started watching for new ooms in manager Jul 23 21:52:01.600994 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.600982 2569 manager.go:349] Starting recovery of all containers Jul 23 21:52:01.601324 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.601302 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 21:52:01.601610 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.601398 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 21:52:01.610148 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.610122 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p84x6" Jul 23 21:52:01.612015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.611998 2569 manager.go:354] Recovery completed Jul 23 21:52:01.616203 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.616188 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.618239 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.618220 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.618307 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.618252 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.618307 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.618265 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.619460 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.619446 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 21:52:01.619460 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.619455 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.619460 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.619459 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 21:52:01.619714 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.619639 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 23 21:52:01.620121 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.620107 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.620177 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.620132 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.620177 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.620142 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.620647 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.620634 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p84x6" Jul 23 21:52:01.621574 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.621563 2569 policy_none.go:49] "None policy: Start" Jul 23 21:52:01.621634 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.621580 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 21:52:01.621634 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.621595 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 21:52:01.622611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.622597 2569 policy_none.go:47] "Start" Jul 23 21:52:01.663148 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663008 2569 manager.go:337] "Starting Device Plugin manager" Jul 23 21:52:01.663248 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.663172 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 21:52:01.663248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663183 2569 server.go:87] "Starting device plugin registration server" Jul 23 21:52:01.663403 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663392 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 21:52:01.663446 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663425 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 21:52:01.663534 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663518 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 21:52:01.663636 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663624 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 21:52:01.663636 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.663634 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 21:52:01.664088 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.664061 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 21:52:01.664142 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.664109 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:01.763960 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.763936 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.765004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.764837 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.765004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.764865 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.765004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.764881 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.765004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.764905 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.769242 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.769222 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 21:52:01.770316 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.770287 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 21:52:01.770316 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.770320 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 21:52:01.770460 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.770339 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 21:52:01.770460 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.770368 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 21:52:01.772833 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.772812 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 21:52:01.775026 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.775012 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.775104 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.775034 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-110.ec2.internal\": node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:01.803502 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.803436 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:01.871489 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.871459 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal"] Jul 23 21:52:01.871567 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.871540 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.873110 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.873096 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.873167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.873127 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.873167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.873141 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.874391 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.874379 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.874509 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.874495 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.874590 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.874535 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.875224 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875208 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.875224 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875221 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.875323 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875235 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.875323 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875242 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.875323 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875247 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.875323 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.875251 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.876849 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.876826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.876947 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.876872 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 21:52:01.877806 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.877789 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Jul 23 21:52:01.877919 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.877813 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 21:52:01.877919 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.877828 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Jul 23 21:52:01.901956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.901934 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d12a7c9f3bb00f63b0202d19fa6945-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"72d12a7c9f3bb00f63b0202d19fa6945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.902038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.901963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.902038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:01.901991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.902038 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.902008 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-110.ec2.internal\" not found" node="ip-10-0-142-110.ec2.internal" Jul 23 21:52:01.903859 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.903840 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:01.906249 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:01.906234 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-110.ec2.internal\" not found" node="ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002616 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d12a7c9f3bb00f63b0202d19fa6945-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"72d12a7c9f3bb00f63b0202d19fa6945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002616 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002694 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d12a7c9f3bb00f63b0202d19fa6945-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"72d12a7c9f3bb00f63b0202d19fa6945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002747 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.002785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.002777 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a410fb572c1ecc3bc886dc4f6feda4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"e2a410fb572c1ecc3bc886dc4f6feda4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.004701 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.004683 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.104841 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.104777 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.205306 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.205286 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.205407 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.205328 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.209459 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.209056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:02.305566 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.305537 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.406143 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.406072 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.506598 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.506572 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.540756 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.540725 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 21:52:02.541125 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.540870 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 21:52:02.580591 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.580566 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 21:52:02.606676 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.606655 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.624820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.624795 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 21:47:01 +0000 UTC" deadline="2028-04-17 18:07:31.639184242 +0000 UTC" Jul 23 21:52:02.624820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.624818 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15212h15m29.014368832s" Jul 23 21:52:02.707419 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.707400 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.808043 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.808017 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:02.815623 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:02.815590 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d12a7c9f3bb00f63b0202d19fa6945.slice/crio-ff1b6a42f56b66cd2ef488c0ecc22fa3d9f8330f9feaac9e1b4830c886b534ee WatchSource:0}: Error finding container ff1b6a42f56b66cd2ef488c0ecc22fa3d9f8330f9feaac9e1b4830c886b534ee: Status 404 returned error can't find the container with id ff1b6a42f56b66cd2ef488c0ecc22fa3d9f8330f9feaac9e1b4830c886b534ee Jul 23 21:52:02.816081 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:02.816055 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a410fb572c1ecc3bc886dc4f6feda4.slice/crio-dd4251fdd969c224fa2c68ffd5fd1883c8b1d4bad28e88d13cdf4b2e36c147cf WatchSource:0}: Error finding container dd4251fdd969c224fa2c68ffd5fd1883c8b1d4bad28e88d13cdf4b2e36c147cf: Status 404 returned error can't find the container with id dd4251fdd969c224fa2c68ffd5fd1883c8b1d4bad28e88d13cdf4b2e36c147cf Jul 23 21:52:02.819829 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.819809 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 21:52:02.823841 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:02.823825 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 21:52:02.908650 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:02.908619 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Jul 23 21:52:03.003360 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.003285 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 21:52:03.100567 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.100539 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Jul 23 21:52:03.109696 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.109679 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 21:52:03.110813 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.110794 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Jul 23 21:52:03.124841 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.124820 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 21:52:03.395834 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.395759 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 21:52:03.582239 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.582207 2569 apiserver.go:52] "Watching apiserver" Jul 23 21:52:03.590799 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.590774 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 21:52:03.593189 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.593160 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-b5sb5","kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q4plw","openshift-dns/node-resolver-h86nk","openshift-image-registry/node-ca-f5t8n","openshift-network-operator/iptables-alerter-txckh","kube-system/konnectivity-agent-p6f5d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-h5km9","openshift-multus/multus-dxqxh","openshift-multus/network-metrics-daemon-cznsg","openshift-network-diagnostics/network-check-target-bh2r8","openshift-ovn-kubernetes/ovnkube-node-mf658"] Jul 23 21:52:03.595731 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.595713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.598018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.597994 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 21:52:03.598295 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.598269 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 21:52:03.598387 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.598291 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 21:52:03.598442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.598285 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cxbjf\"" Jul 23 21:52:03.598442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.598404 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.598914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.598894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.599893 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.599785 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 21:52:03.600294 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.600277 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.601896 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.601620 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c8jk\"" Jul 23 21:52:03.601896 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.601846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.602034 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.601920 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 21:52:03.602034 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.602013 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:03.602125 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.602083 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:03.602586 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.602460 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.602586 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.602541 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 21:52:03.602586 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.602558 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 21:52:03.602586 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.602459 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 21:52:03.605052 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.605001 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.605142 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.605113 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:03.607566 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.607223 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.609306 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609288 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.609592 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609575 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.609692 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609672 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.609785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.609874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-slash\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.609874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-ovn\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.609874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-script-lib\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24k7\" (UniqueName: \"kubernetes.io/projected/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-kube-api-access-g24k7\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609909 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-system-cni-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-systemd-units\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.609979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-netns\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-bin\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610043 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 21:52:03.610068 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610096 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5twfc\"" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-etc-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovn-node-metrics-cert\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610218 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xpl\" (UniqueName: \"kubernetes.io/projected/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-kube-api-access-c4xpl\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610242 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-systemd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-var-lib-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610285 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-node-log\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-log-socket\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610377 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-config\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-os-release\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-kubelet\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.610558 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-netd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.611235 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610561 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-env-overrides\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.611235 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.610616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cnibin\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.611778 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.611760 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.612340 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.612324 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.612757 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.612533 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 21:52:03.612757 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.612557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-krv2k\"" Jul 23 21:52:03.612886 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.612810 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.614020 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.613999 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 21:52:03.614113 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.614043 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l7dh9\"" Jul 23 21:52:03.614285 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.614269 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.616539 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.616521 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.616808 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.616782 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.617197 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.616996 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 21:52:03.617197 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.617016 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ffn2s\"" Jul 23 21:52:03.617197 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.617174 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.619452 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.619434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.620003 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.619851 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.620003 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.619887 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4v5r\"" Jul 23 21:52:03.620003 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.619894 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.621496 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.621382 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.621496 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.621447 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:03.622639 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.622224 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 21:52:03.622639 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.622282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9n7p2\"" Jul 23 21:52:03.622639 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.622320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 21:52:03.623837 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.623815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.626364 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.626217 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 21:52:03.626364 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.626216 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c7796\"" Jul 23 21:52:03.626521 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.626445 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 21:52:03.632567 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.632545 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xb754" Jul 23 21:52:03.641766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.641712 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xb754" Jul 23 21:52:03.701440 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.701414 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 21:52:03.711471 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-daemon-config\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.711603 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-systemd-units\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.711603 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-conf\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.711603 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjllb\" (UniqueName: \"kubernetes.io/projected/ac541a93-e7bc-4340-ae3b-e87077adcb31-kube-api-access-rjllb\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.711603 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-socket-dir-parent\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-systemd-units\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjxb\" (UniqueName: \"kubernetes.io/projected/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-kube-api-access-9bjxb\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711757 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-kubernetes\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdsl\" (UniqueName: \"kubernetes.io/projected/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kube-api-access-tbdsl\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6869373b-5e3c-428e-bd23-e8592620ef52-agent-certs\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.711866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-etc-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.711980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-etc-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xpl\" (UniqueName: \"kubernetes.io/projected/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-kube-api-access-c4xpl\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-systemd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-node-log\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-host\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-kubelet\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712139 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-systemd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506af321-bdfe-4bb8-b533-65c6bf9ebb69-host-slash\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6869373b-5e3c-428e-bd23-e8592620ef52-konnectivity-ca\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-os-release\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-env-overrides\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4mtp\" (UniqueName: \"kubernetes.io/projected/ee851f90-0e5c-4feb-accf-1128ed368f40-kube-api-access-n4mtp\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-registration-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-sys-fs\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.712423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cnibin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712438 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-node-log\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712105 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712462 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-ovn\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-script-lib\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712599 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712724 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-os-release\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712805 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-run-ovn\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712927 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-env-overrides\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-tmp\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713009 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.712997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-etc-selinux\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-kubelet-config\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-systemd\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dr97\" (UniqueName: \"kubernetes.io/projected/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-kube-api-access-9dr97\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.713234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-netns\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cni-binary-copy\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713242 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-netns\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-bin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-bin\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-socket-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713358 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-script-lib\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/506af321-bdfe-4bb8-b533-65c6bf9ebb69-iptables-alerter-script\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-system-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713405 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovn-node-metrics-cert\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713465 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-bin\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713476 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-modprobe-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-var-lib-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.713831 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-run\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz4n\" (UniqueName: \"kubernetes.io/projected/235d6a13-4446-4a06-93e4-ae7e9a3634c9-kube-api-access-jvz4n\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-hostroot\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-conf-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-var-lib-openvswitch\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-multus-certs\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713809 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-log-socket\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713791 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-config\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-log-socket\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.713916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysconfig\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-tuned\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c5b\" (UniqueName: \"kubernetes.io/projected/506af321-bdfe-4bb8-b533-65c6bf9ebb69-kube-api-access-52c5b\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-multus\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.714621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-etc-kubernetes\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714199 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-kubelet\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-netd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-kubelet\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee851f90-0e5c-4feb-accf-1128ed368f40-host\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714341 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-cni-netd\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714372 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovnkube-config\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714375 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-device-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-hosts-file\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-os-release\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-netns\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cnibin\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-slash\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714549 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24k7\" (UniqueName: \"kubernetes.io/projected/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-kube-api-access-g24k7\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-sys\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-cnibin\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-lib-modules\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.715381 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-host-slash\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714604 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-tmp-dir\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-dbus\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-system-cni-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-var-lib-kubelet\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee851f90-0e5c-4feb-accf-1128ed368f40-serviceca\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714724 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-system-cni-dir\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.716175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.714742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-k8s-cni-cncf-io\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.717945 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.717922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-ovn-node-metrics-cert\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.728272 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.728249 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:03.728272 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.728272 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:03.728400 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.728282 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:03.728400 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.728331 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:04.228316819 +0000 UTC m=+4.253650352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:03.729363 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.729340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xpl\" (UniqueName: \"kubernetes.io/projected/439b24d5-9bad-4c69-9e1c-4bb7c94631d0-kube-api-access-c4xpl\") pod \"multus-additional-cni-plugins-h5km9\" (UID: \"439b24d5-9bad-4c69-9e1c-4bb7c94631d0\") " pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.731449 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.731426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24k7\" (UniqueName: \"kubernetes.io/projected/5fd867b5-4c02-41fb-9b89-c95297ed5dd3-kube-api-access-g24k7\") pod \"ovnkube-node-mf658\" (UID: \"5fd867b5-4c02-41fb-9b89-c95297ed5dd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.775510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.775444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"e2a410fb572c1ecc3bc886dc4f6feda4","Type":"ContainerStarted","Data":"dd4251fdd969c224fa2c68ffd5fd1883c8b1d4bad28e88d13cdf4b2e36c147cf"} Jul 23 21:52:03.776673 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.776644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"72d12a7c9f3bb00f63b0202d19fa6945","Type":"ContainerStarted","Data":"ff1b6a42f56b66cd2ef488c0ecc22fa3d9f8330f9feaac9e1b4830c886b534ee"} Jul 23 21:52:03.815010 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.814980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-socket-dir-parent\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjxb\" (UniqueName: \"kubernetes.io/projected/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-kube-api-access-9bjxb\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-kubernetes\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815143 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-kubernetes\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdsl\" (UniqueName: \"kubernetes.io/projected/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kube-api-access-tbdsl\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6869373b-5e3c-428e-bd23-e8592620ef52-agent-certs\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.815248 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-socket-dir-parent\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-host\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-kubelet\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506af321-bdfe-4bb8-b533-65c6bf9ebb69-host-slash\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815347 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-host\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6869373b-5e3c-428e-bd23-e8592620ef52-konnectivity-ca\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815389 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-kubelet\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815411 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4mtp\" (UniqueName: \"kubernetes.io/projected/ee851f90-0e5c-4feb-accf-1128ed368f40-kube-api-access-n4mtp\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815416 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506af321-bdfe-4bb8-b533-65c6bf9ebb69-host-slash\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-registration-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-sys-fs\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cnibin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-tmp\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-registration-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-etc-selinux\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.815611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-kubelet-config\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815619 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cnibin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-systemd\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-etc-selinux\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dr97\" (UniqueName: \"kubernetes.io/projected/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-kube-api-access-9dr97\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815681 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-systemd\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cni-binary-copy\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815700 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-kubelet-config\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-sys-fs\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-bin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-socket-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/506af321-bdfe-4bb8-b533-65c6bf9ebb69-iptables-alerter-script\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-bin\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-system-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-modprobe-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815951 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-socket-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-run\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.816389 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.815989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz4n\" (UniqueName: \"kubernetes.io/projected/235d6a13-4446-4a06-93e4-ae7e9a3634c9-kube-api-access-jvz4n\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6869373b-5e3c-428e-bd23-e8592620ef52-konnectivity-ca\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816089 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-run\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-system-cni-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816146 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-modprobe-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-hostroot\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-cni-binary-copy\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-hostroot\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-conf-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-multus-certs\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysconfig\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816260 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-conf-dir\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-tuned\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c5b\" (UniqueName: \"kubernetes.io/projected/506af321-bdfe-4bb8-b533-65c6bf9ebb69-kube-api-access-52c5b\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816348 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysconfig\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816370 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/506af321-bdfe-4bb8-b533-65c6bf9ebb69-iptables-alerter-script\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816372 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.817276 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816326 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-multus-certs\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-multus\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-var-lib-cni-multus\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816438 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-etc-kubernetes\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee851f90-0e5c-4feb-accf-1128ed368f40-host\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-device-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-hosts-file\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816875 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-etc-kubernetes\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.816558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee851f90-0e5c-4feb-accf-1128ed368f40-host\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817166 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-os-release\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-netns\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-sys\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-lib-modules\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-tmp-dir\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-dbus\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-var-lib-kubelet\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee851f90-0e5c-4feb-accf-1128ed368f40-serviceca\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.818058 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-k8s-cni-cncf-io\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-daemon-config\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-conf\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjllb\" (UniqueName: \"kubernetes.io/projected/ac541a93-e7bc-4340-ae3b-e87077adcb31-kube-api-access-rjllb\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.817810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.817940 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.818012 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:04.317997099 +0000 UTC m=+4.343330626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818110 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-tmp\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818236 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-k8s-cni-cncf-io\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818343 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6869373b-5e3c-428e-bd23-e8592620ef52-agent-certs\") pod \"konnectivity-agent-p6f5d\" (UID: \"6869373b-5e3c-428e-bd23-e8592620ef52\") " pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818368 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-tuned\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-var-lib-kubelet\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.818506 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818512 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-dbus\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818515 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-tmp-dir\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-d\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.818874 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:03.818654 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:04.318633104 +0000 UTC m=+4.343966621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818725 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-os-release\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818740 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-lib-modules\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-etc-sysctl-conf\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818777 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-device-dir\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818825 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-hosts-file\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818871 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-host-run-netns\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818904 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-multus-daemon-config\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.818912 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac541a93-e7bc-4340-ae3b-e87077adcb31-sys\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.819668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.819205 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee851f90-0e5c-4feb-accf-1128ed368f40-serviceca\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.824886 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.824856 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz4n\" (UniqueName: \"kubernetes.io/projected/235d6a13-4446-4a06-93e4-ae7e9a3634c9-kube-api-access-jvz4n\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:03.825031 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.825012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjxb\" (UniqueName: \"kubernetes.io/projected/211b0844-c2e3-4ad6-81ba-014d94ed7eb4-kube-api-access-9bjxb\") pod \"multus-dxqxh\" (UID: \"211b0844-c2e3-4ad6-81ba-014d94ed7eb4\") " pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.825099 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.825012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4mtp\" (UniqueName: \"kubernetes.io/projected/ee851f90-0e5c-4feb-accf-1128ed368f40-kube-api-access-n4mtp\") pod \"node-ca-f5t8n\" (UID: \"ee851f90-0e5c-4feb-accf-1128ed368f40\") " pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.825099 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.825038 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdsl\" (UniqueName: \"kubernetes.io/projected/4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54-kube-api-access-tbdsl\") pod \"aws-ebs-csi-driver-node-9r9dx\" (UID: \"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.825860 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.825838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c5b\" (UniqueName: \"kubernetes.io/projected/506af321-bdfe-4bb8-b533-65c6bf9ebb69-kube-api-access-52c5b\") pod \"iptables-alerter-txckh\" (UID: \"506af321-bdfe-4bb8-b533-65c6bf9ebb69\") " pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.826312 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.826291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dr97\" (UniqueName: \"kubernetes.io/projected/b6c64ade-a26a-4e1e-b1eb-776447b72dc5-kube-api-access-9dr97\") pod \"node-resolver-h86nk\" (UID: \"b6c64ade-a26a-4e1e-b1eb-776447b72dc5\") " pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.833390 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.833373 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjllb\" (UniqueName: \"kubernetes.io/projected/ac541a93-e7bc-4340-ae3b-e87077adcb31-kube-api-access-rjllb\") pod \"tuned-q4plw\" (UID: \"ac541a93-e7bc-4340-ae3b-e87077adcb31\") " pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:03.911150 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.911065 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h5km9" Jul 23 21:52:03.917732 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.917707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:03.927372 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.927349 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5t8n" Jul 23 21:52:03.935993 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.935976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txckh" Jul 23 21:52:03.942560 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.942539 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxqxh" Jul 23 21:52:03.951146 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.951124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" Jul 23 21:52:03.958638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.958620 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h86nk" Jul 23 21:52:03.965220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.965201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:03.972004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:03.971984 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4plw" Jul 23 21:52:04.321739 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.321647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:04.321739 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.321695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.321756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321837 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321885 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321908 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:05.321890514 +0000 UTC m=+5.347224049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321908 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321930 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321952 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:04.321969 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.321964 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:05.32195504 +0000 UTC m=+5.347288577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:04.322313 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:04.322002 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:05.321986698 +0000 UTC m=+5.347320245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:04.645065 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.644976 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 21:47:03 +0000 UTC" deadline="2028-03-03 00:08:40.672246976 +0000 UTC" Jul 23 21:52:04.645065 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.645023 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14114h16m36.027226381s" Jul 23 21:52:04.726325 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.726302 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0c5625_60bb_4c0e_90cd_fcb7ff70cc54.slice/crio-ac377f049ff9df9e56c9ef9b7db585b680b8c1bbdd1784b52551df7b1b40e946 WatchSource:0}: Error finding container ac377f049ff9df9e56c9ef9b7db585b680b8c1bbdd1784b52551df7b1b40e946: Status 404 returned error can't find the container with id ac377f049ff9df9e56c9ef9b7db585b680b8c1bbdd1784b52551df7b1b40e946 Jul 23 21:52:04.727734 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.727622 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c64ade_a26a_4e1e_b1eb_776447b72dc5.slice/crio-bed2272d9cf071ee3f4032eb5cb7a0e0be4682ecceded3470c0461a801315e0e WatchSource:0}: Error finding container bed2272d9cf071ee3f4032eb5cb7a0e0be4682ecceded3470c0461a801315e0e: Status 404 returned error can't find the container with id bed2272d9cf071ee3f4032eb5cb7a0e0be4682ecceded3470c0461a801315e0e Jul 23 21:52:04.731910 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.731890 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439b24d5_9bad_4c69_9e1c_4bb7c94631d0.slice/crio-5412858dc1ae260e1ddc7bb92b84050f6a2c9a1d20afedd9090ede43baac134e WatchSource:0}: Error finding container 5412858dc1ae260e1ddc7bb92b84050f6a2c9a1d20afedd9090ede43baac134e: Status 404 returned error can't find the container with id 5412858dc1ae260e1ddc7bb92b84050f6a2c9a1d20afedd9090ede43baac134e Jul 23 21:52:04.732380 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.732363 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6869373b_5e3c_428e_bd23_e8592620ef52.slice/crio-6e555d361a01a12d864c9182a80ae718da3f0635e6348954a0e4ad43b3dcd1d1 WatchSource:0}: Error finding container 6e555d361a01a12d864c9182a80ae718da3f0635e6348954a0e4ad43b3dcd1d1: Status 404 returned error can't find the container with id 6e555d361a01a12d864c9182a80ae718da3f0635e6348954a0e4ad43b3dcd1d1 Jul 23 21:52:04.733031 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.732995 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd867b5_4c02_41fb_9b89_c95297ed5dd3.slice/crio-ea93cbd4c3b23673c4aae817c1e8b7d1448df14b14b35c63c0bbb2362c9e4765 WatchSource:0}: Error finding container ea93cbd4c3b23673c4aae817c1e8b7d1448df14b14b35c63c0bbb2362c9e4765: Status 404 returned error can't find the container with id ea93cbd4c3b23673c4aae817c1e8b7d1448df14b14b35c63c0bbb2362c9e4765 Jul 23 21:52:04.739406 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.739376 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac541a93_e7bc_4340_ae3b_e87077adcb31.slice/crio-b7821e9764407bd77b058b13ed038528f6c27777527881da8610005526a0e8ab WatchSource:0}: Error finding container b7821e9764407bd77b058b13ed038528f6c27777527881da8610005526a0e8ab: Status 404 returned error can't find the container with id b7821e9764407bd77b058b13ed038528f6c27777527881da8610005526a0e8ab Jul 23 21:52:04.740274 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:04.740254 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506af321_bdfe_4bb8_b533_65c6bf9ebb69.slice/crio-4e98bd66893610cdd0b06c5f3e25746836142b005964701e5fee55844acb750d WatchSource:0}: Error finding container 4e98bd66893610cdd0b06c5f3e25746836142b005964701e5fee55844acb750d: Status 404 returned error can't find the container with id 4e98bd66893610cdd0b06c5f3e25746836142b005964701e5fee55844acb750d Jul 23 21:52:04.783522 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.783469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerStarted","Data":"5412858dc1ae260e1ddc7bb92b84050f6a2c9a1d20afedd9090ede43baac134e"} Jul 23 21:52:04.784620 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.784596 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5t8n" event={"ID":"ee851f90-0e5c-4feb-accf-1128ed368f40","Type":"ContainerStarted","Data":"72de7570b794bc9fcf2c37de29fb1ee5a329d97a17676550b325ed4f4fe078a7"} Jul 23 21:52:04.785666 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.785647 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"ea93cbd4c3b23673c4aae817c1e8b7d1448df14b14b35c63c0bbb2362c9e4765"} Jul 23 21:52:04.786658 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.786636 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h86nk" event={"ID":"b6c64ade-a26a-4e1e-b1eb-776447b72dc5","Type":"ContainerStarted","Data":"bed2272d9cf071ee3f4032eb5cb7a0e0be4682ecceded3470c0461a801315e0e"} Jul 23 21:52:04.787840 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.787814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" event={"ID":"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54","Type":"ContainerStarted","Data":"ac377f049ff9df9e56c9ef9b7db585b680b8c1bbdd1784b52551df7b1b40e946"} Jul 23 21:52:04.789020 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.789001 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txckh" event={"ID":"506af321-bdfe-4bb8-b533-65c6bf9ebb69","Type":"ContainerStarted","Data":"4e98bd66893610cdd0b06c5f3e25746836142b005964701e5fee55844acb750d"} Jul 23 21:52:04.790123 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.790104 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxqxh" event={"ID":"211b0844-c2e3-4ad6-81ba-014d94ed7eb4","Type":"ContainerStarted","Data":"6ecbd22549d0899e49ae4b0f953e61f0c483ddf48bddc0c7633072411d140e47"} Jul 23 21:52:04.791281 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.791257 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4plw" event={"ID":"ac541a93-e7bc-4340-ae3b-e87077adcb31","Type":"ContainerStarted","Data":"b7821e9764407bd77b058b13ed038528f6c27777527881da8610005526a0e8ab"} Jul 23 21:52:04.792275 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:04.792255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6f5d" event={"ID":"6869373b-5e3c-428e-bd23-e8592620ef52","Type":"ContainerStarted","Data":"6e555d361a01a12d864c9182a80ae718da3f0635e6348954a0e4ad43b3dcd1d1"} Jul 23 21:52:05.329347 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.329104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:05.329347 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.329331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:05.329600 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.329360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:05.329600 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329553 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:05.329699 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329621 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:07.329603314 +0000 UTC m=+7.354936828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:05.329759 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329721 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:05.329759 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329739 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:05.329759 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329751 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:05.329890 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329788 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:07.329775903 +0000 UTC m=+7.355109419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:05.329890 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329851 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:05.329890 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.329882 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:07.32987132 +0000 UTC m=+7.355204833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:05.645647 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.645537 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 21:47:03 +0000 UTC" deadline="2028-02-16 16:05:43.006523714 +0000 UTC" Jul 23 21:52:05.645647 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.645596 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13746h13m37.360932358s" Jul 23 21:52:05.773616 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.773586 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:05.773775 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.773727 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:05.774216 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.774198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:05.774313 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.774295 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:05.774369 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.774357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:05.774438 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:05.774420 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:05.803906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.803872 2569 generic.go:356] "Generic (PLEG): container finished" podID="e2a410fb572c1ecc3bc886dc4f6feda4" containerID="e3b3f01c1070f4f94b113e72b9369551f33fce7e1627e0ee62b7aaff5ec5d694" exitCode=0 Jul 23 21:52:05.804059 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.803962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"e2a410fb572c1ecc3bc886dc4f6feda4","Type":"ContainerDied","Data":"e3b3f01c1070f4f94b113e72b9369551f33fce7e1627e0ee62b7aaff5ec5d694"} Jul 23 21:52:05.830879 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:05.830769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"72d12a7c9f3bb00f63b0202d19fa6945","Type":"ContainerStarted","Data":"8b6a2e2f26880073d84286f8793355bd4245f58d7df5d520f04108c99ac32082"} Jul 23 21:52:06.861021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:06.860986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"e2a410fb572c1ecc3bc886dc4f6feda4","Type":"ContainerStarted","Data":"bc711d22c24a44bde1d51c261595e426f09f8ef92e775e8fc371fc510250e80b"} Jul 23 21:52:06.878724 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:06.878216 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" podStartSLOduration=3.878196171 podStartE2EDuration="3.878196171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:05.847905699 +0000 UTC m=+5.873239236" watchObservedRunningTime="2026-07-23 21:52:06.878196171 +0000 UTC m=+6.903529708" Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.346544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.346617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.346647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.346782 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.346842 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:11.346824617 +0000 UTC m=+11.372158135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347234 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347253 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347264 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347308 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:11.3472944 +0000 UTC m=+11.372627918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347364 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:07.347492 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.347399 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:11.347388094 +0000 UTC m=+11.372721611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.772509 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.772663 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.772750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.772865 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:07.772933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:07.773098 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:07.773013 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:09.771127 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:09.771096 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:09.771571 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:09.771229 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:09.771571 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:09.771385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:09.771571 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:09.771474 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:09.771571 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:09.771545 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:09.771792 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:09.771635 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.381741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.381806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.381833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.381973 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382030 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:19.382011343 +0000 UTC m=+19.407344871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382202 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382258 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:19.382243639 +0000 UTC m=+19.407577169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382283 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382320 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:11.382352 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382331 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:11.383227 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.382393 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:19.382375202 +0000 UTC m=+19.407708735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:11.771553 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.771463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:11.771695 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.771606 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:11.772148 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.772006 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:11.772148 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.772113 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:11.772324 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:11.772163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:11.772324 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:11.772229 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:13.771121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:13.771142 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:13.771258 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:13.771371 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:13.771535 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:13.773136 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:13.771612 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:15.773937 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:15.773901 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:15.774371 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:15.774035 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:15.774371 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:15.774041 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:15.774371 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:15.774047 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:15.774371 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:15.774152 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:15.774371 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:15.774235 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:17.773375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:17.773334 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:17.773833 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:17.773432 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:17.773833 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:17.773442 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:17.773833 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:17.773544 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:17.773833 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:17.773582 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:17.773833 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:17.773654 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:19.448137 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.448102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.448169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.448188 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448285 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448289 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448310 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448313 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448370 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448354 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:52:35.448334747 +0000 UTC m=+35.473668307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448434 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:35.448417101 +0000 UTC m=+35.473750622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:19.448659 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.448453 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:35.448445662 +0000 UTC m=+35.473779179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:19.774967 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.774890 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:19.775122 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.775013 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:19.775244 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.775222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:19.775323 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:19.775265 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:19.775370 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.775322 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:19.775410 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:19.775375 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:21.771708 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:21.771675 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:21.772190 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:21.771754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:21.772190 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:21.771781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:21.772190 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:21.771805 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:21.772190 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:21.771869 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:21.772190 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:21.771932 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:23.770965 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.770576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:23.771994 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.770606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:23.771994 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:23.771086 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:23.771994 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.770685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:23.771994 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:23.771114 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:23.771994 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:23.771226 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:23.889443 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.889419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h86nk" event={"ID":"b6c64ade-a26a-4e1e-b1eb-776447b72dc5","Type":"ContainerStarted","Data":"5ff188f609276cf9b7e009957c162a7358776a19c8ee1bc8dd1b09427ced30a8"} Jul 23 21:52:23.891205 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.891180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" event={"ID":"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54","Type":"ContainerStarted","Data":"e446d35fc5a78a53ed9865165a7824c02980f1eb676471dffbcf48d4f7e618ae"} Jul 23 21:52:23.894865 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.894651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxqxh" event={"ID":"211b0844-c2e3-4ad6-81ba-014d94ed7eb4","Type":"ContainerStarted","Data":"53e5e17f0c944c96fbf5774aef13cb6b50f48e7f3cd19067ab996272611bc18e"} Jul 23 21:52:23.896900 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.896878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4plw" event={"ID":"ac541a93-e7bc-4340-ae3b-e87077adcb31","Type":"ContainerStarted","Data":"197f84f05a0e4de226e9c50b77fd6858b83e53de16fd2f0fbc26773fd9f1fb81"} Jul 23 21:52:23.898606 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.898587 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6f5d" event={"ID":"6869373b-5e3c-428e-bd23-e8592620ef52","Type":"ContainerStarted","Data":"8f842830c5aaa86ba9285cc5b1ee7d438abd4add0b7fcf4c729bd7edb92588f4"} Jul 23 21:52:23.900188 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.900164 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="7a658a66c62414a3cd5623b76c1f6a5a76c81069c20347a3451e760be582eebe" exitCode=0 Jul 23 21:52:23.900310 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.900240 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"7a658a66c62414a3cd5623b76c1f6a5a76c81069c20347a3451e760be582eebe"} Jul 23 21:52:23.902639 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.902475 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5t8n" event={"ID":"ee851f90-0e5c-4feb-accf-1128ed368f40","Type":"ContainerStarted","Data":"3efcb994739bc43b3c7009fc98f9cfa86edf076f28ce5e2ed932aa43bf26e868"} Jul 23 21:52:23.905457 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.905436 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:52:23.905885 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.905860 2569 generic.go:356] "Generic (PLEG): container finished" podID="5fd867b5-4c02-41fb-9b89-c95297ed5dd3" containerID="efab54ac88fbc6fb666f154bf6da3bf0130499d4ca5a95a66e8b340496f0d882" exitCode=1 Jul 23 21:52:23.905973 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.905896 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"c251ebaeae0bf10d39cb932fe7f7da71ba3ffb099604607f6ba6bf0a2e2f7a2c"} Jul 23 21:52:23.905973 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.905961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"6707c9f42a6a05eb35e7b6943af1db2051049e2463d7bc8ce32cd8ccbe5b5d84"} Jul 23 21:52:23.906093 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.906007 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"7c5d9c663e4ed1065b26ddcc5782af366f90423029171efcb698c8cdc8f049b8"} Jul 23 21:52:23.906093 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.906054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerDied","Data":"efab54ac88fbc6fb666f154bf6da3bf0130499d4ca5a95a66e8b340496f0d882"} Jul 23 21:52:23.906194 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.906100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"0f97c22152c19bdf21f0f373382c992d58bcc235e6928061d674f209178aa105"} Jul 23 21:52:23.906362 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.906320 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" podStartSLOduration=20.906304978 podStartE2EDuration="20.906304978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:06.879105847 +0000 UTC m=+6.904439382" watchObservedRunningTime="2026-07-23 21:52:23.906304978 +0000 UTC m=+23.931638514" Jul 23 21:52:23.907066 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.907027 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h86nk" podStartSLOduration=3.473848113 podStartE2EDuration="21.90701837s" totalImagesPullingTime="18.433170257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:23.906428727 +0000 UTC m=+23.931762253" watchObservedRunningTime="2026-07-23 21:52:23.90701837 +0000 UTC m=+23.932351905" Jul 23 21:52:23.924369 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.924299 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4plw" podStartSLOduration=3.458070695 podStartE2EDuration="21.924274722s" totalImagesPullingTime="18.466204027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:23.923810843 +0000 UTC m=+23.949144379" watchObservedRunningTime="2026-07-23 21:52:23.924274722 +0000 UTC m=+23.949608258" Jul 23 21:52:23.939295 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.939249 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f5t8n" podStartSLOduration=4.513655659 podStartE2EDuration="22.939232538s" totalImagesPullingTime="18.425576879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:23.939049042 +0000 UTC m=+23.964382578" watchObservedRunningTime="2026-07-23 21:52:23.939232538 +0000 UTC m=+23.964566074" Jul 23 21:52:23.970795 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.970752 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dxqxh" podStartSLOduration=4.134480225 podStartE2EDuration="22.970737322s" totalImagesPullingTime="18.836257097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:23.95546028 +0000 UTC m=+23.980793816" watchObservedRunningTime="2026-07-23 21:52:23.970737322 +0000 UTC m=+23.996070860" Jul 23 21:52:23.993963 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:23.993910 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p6f5d" podStartSLOduration=11.735405983 podStartE2EDuration="21.993893124s" totalImagesPullingTime="10.258487141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:23.970796846 +0000 UTC m=+23.996130382" watchObservedRunningTime="2026-07-23 21:52:23.993893124 +0000 UTC m=+24.019226659" Jul 23 21:52:24.681126 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:24.681084 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 21:52:24.911502 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:24.911404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:52:24.911986 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:24.911851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"18bd43f261393bba893726acba9480d8462f0ae8a9977f54dc765be0ff4f53e2"} Jul 23 21:52:24.913879 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:24.913851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" event={"ID":"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54","Type":"ContainerStarted","Data":"0f2644b44f7451180d200f1c3cad3d934b7ec3de63728dc09f66a4a453869483"} Jul 23 21:52:24.915533 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:24.915507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txckh" event={"ID":"506af321-bdfe-4bb8-b533-65c6bf9ebb69","Type":"ContainerStarted","Data":"c8190d52d365dc1bbfd23334db59a96a8d2f8cbe7bba7ae7118c2bb72c3b08ab"} Jul 23 21:52:25.676924 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.676826 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T21:52:24.68110723Z","UUID":"e63cfe5f-f78c-42fa-a600-e0054e85db49","Handler":null,"Name":"","Endpoint":""} Jul 23 21:52:25.678538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.678518 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 21:52:25.678657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.678553 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 21:52:25.771010 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.770979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:25.771207 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.770981 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:25.771207 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:25.771113 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:25.771207 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.771135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:25.771369 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:25.771248 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:25.771369 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:25.771329 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:25.923210 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.923169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" event={"ID":"4a0c5625-60bb-4c0e-90cd-fcb7ff70cc54","Type":"ContainerStarted","Data":"f560d1142d1d5fce5a0ae028aaf3754f89df59e485a8b6bc2fe2093e49684924"} Jul 23 21:52:25.944894 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.944825 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-txckh" podStartSLOduration=6.52403633 podStartE2EDuration="24.944811947s" totalImagesPullingTime="18.420775617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:24.935665636 +0000 UTC m=+24.960999170" watchObservedRunningTime="2026-07-23 21:52:25.944811947 +0000 UTC m=+25.970145482" Jul 23 21:52:25.945004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:25.944923 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9r9dx" podStartSLOduration=4.286300135 podStartE2EDuration="24.944918709s" totalImagesPullingTime="20.658618574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:25.944336124 +0000 UTC m=+25.969669661" watchObservedRunningTime="2026-07-23 21:52:25.944918709 +0000 UTC m=+25.970252244" Jul 23 21:52:26.928825 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:26.928800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:52:26.929223 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:26.929105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"66a74f05c7b2e2e4fb37077859f3054024fdb6ace3157181051530fa6dae0341"} Jul 23 21:52:27.711208 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.711174 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:27.711741 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.711718 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:27.771134 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.771105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:27.771282 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.771105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:27.771282 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.771220 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:27.771282 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:27.771216 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:27.771429 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:27.771284 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:27.771429 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:27.771400 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:27.931427 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.931217 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:27.931859 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:27.931798 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p6f5d" Jul 23 21:52:28.934986 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.934957 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="0672f8e612c50d72e78993cee01441a696d4b67c9d65a0e94822d4abce92a62c" exitCode=0 Jul 23 21:52:28.935857 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.935031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"0672f8e612c50d72e78993cee01441a696d4b67c9d65a0e94822d4abce92a62c"} Jul 23 21:52:28.938466 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.938453 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:52:28.938866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.938843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"4f8eaf3abf44c2a1e070542cd84e11ca9e021fec1fc17688ec6b6f1c5a436846"} Jul 23 21:52:28.942072 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.939623 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:28.942072 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.939909 2569 scope.go:117] "RemoveContainer" containerID="efab54ac88fbc6fb666f154bf6da3bf0130499d4ca5a95a66e8b340496f0d882" Jul 23 21:52:28.942072 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.940383 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:28.958752 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:28.958660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:29.770511 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.770470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:29.770755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.770495 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:29.770755 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:29.770602 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:29.770755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.770496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:29.770755 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:29.770647 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:29.770755 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:29.770718 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:29.944221 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.944193 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:52:29.944632 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.944603 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" event={"ID":"5fd867b5-4c02-41fb-9b89-c95297ed5dd3","Type":"ContainerStarted","Data":"2fc8f85864e1fc0037f82cdb31dae3b7e83d2fa9e76af265dcdd58617bef0fa2"} Jul 23 21:52:29.944845 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.944832 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:29.958818 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.958793 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:52:29.974538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:29.974497 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" podStartSLOduration=10.455354781 podStartE2EDuration="28.974467083s" totalImagesPullingTime="18.519112302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:29.973852657 +0000 UTC m=+29.999186190" watchObservedRunningTime="2026-07-23 21:52:29.974467083 +0000 UTC m=+29.999800617" Jul 23 21:52:30.949093 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:30.949056 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="74c4eaf4ae29d274fcab024b0296e35ec11ccfa96a76e63f5adc0b34483bc03d" exitCode=0 Jul 23 21:52:30.949543 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:30.949145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"74c4eaf4ae29d274fcab024b0296e35ec11ccfa96a76e63f5adc0b34483bc03d"} Jul 23 21:52:31.771355 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:31.771320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:31.771530 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:31.771410 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:31.771530 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:31.771508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:31.771627 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:31.771596 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:31.771667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:31.771648 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:31.771736 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:31.771720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:32.960563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:32.960531 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="3a4cd8a5e6aeb22725ce8bb2486407d63b22547b5878a22e99c636c937a81ea9" exitCode=0 Jul 23 21:52:32.960923 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:32.960579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"3a4cd8a5e6aeb22725ce8bb2486407d63b22547b5878a22e99c636c937a81ea9"} Jul 23 21:52:33.771495 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:33.771459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:33.771671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:33.771459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:33.771671 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:33.771607 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:33.771671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:33.771470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:33.771831 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:33.771670 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:33.771831 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:33.771742 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:34.511705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.511670 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh2r8"] Jul 23 21:52:34.512100 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.511788 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:34.512100 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:34.511922 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:34.512417 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.512394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b5sb5"] Jul 23 21:52:34.512541 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.512494 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:34.512607 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:34.512593 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:34.515264 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.515241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cznsg"] Jul 23 21:52:34.515360 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:34.515322 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:34.515537 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:34.515412 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:35.467827 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:35.467787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:35.467866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:35.467894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.467959 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.467978 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.467986 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.467990 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.467999 2569 projected.go:196] Error preparing data for projected volume kube-api-access-nvbnr for pod openshift-network-diagnostics/network-check-target-bh2r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.468031 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret podName:1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da nodeName:}" failed. No retries permitted until 2026-07-23 21:53:07.468015528 +0000 UTC m=+67.493349044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret") pod "global-pull-secret-syncer-b5sb5" (UID: "1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da") : object "kube-system"/"original-pull-secret" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.468050 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs podName:235d6a13-4446-4a06-93e4-ae7e9a3634c9 nodeName:}" failed. No retries permitted until 2026-07-23 21:53:07.468042655 +0000 UTC m=+67.493376168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs") pod "network-metrics-daemon-cznsg" (UID: "235d6a13-4446-4a06-93e4-ae7e9a3634c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 21:52:35.468078 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.468062 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr podName:a4d14601-2e8f-4cb4-bf07-6b9f72091874 nodeName:}" failed. No retries permitted until 2026-07-23 21:53:07.468055057 +0000 UTC m=+67.493388574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvbnr" (UniqueName: "kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr") pod "network-check-target-bh2r8" (UID: "a4d14601-2e8f-4cb4-bf07-6b9f72091874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 21:52:35.771135 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:35.771052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:35.771543 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:35.771052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:35.771543 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.771190 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:35.771543 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:35.771226 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:36.771157 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:36.770944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:36.771520 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:36.771201 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh2r8" podUID="a4d14601-2e8f-4cb4-bf07-6b9f72091874" Jul 23 21:52:36.972493 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:36.972458 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="dd5389be0dd0d46aac252cec518067fbd2b33928e52af8bdaff89c93821ad0ff" exitCode=0 Jul 23 21:52:36.972585 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:36.972512 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"dd5389be0dd0d46aac252cec518067fbd2b33928e52af8bdaff89c93821ad0ff"} Jul 23 21:52:37.771240 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:37.771207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:37.771675 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:37.771210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:37.771675 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:37.771309 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cznsg" podUID="235d6a13-4446-4a06-93e4-ae7e9a3634c9" Jul 23 21:52:37.771675 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:37.771412 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5sb5" podUID="1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da" Jul 23 21:52:37.977571 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:37.977535 2569 generic.go:356] "Generic (PLEG): container finished" podID="439b24d5-9bad-4c69-9e1c-4bb7c94631d0" containerID="9b0d3b99638a0df95a19921f2c3ef6012152852a0fc088f92e7da581f8cb7fcf" exitCode=0 Jul 23 21:52:37.977704 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:37.977598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerDied","Data":"9b0d3b99638a0df95a19921f2c3ef6012152852a0fc088f92e7da581f8cb7fcf"} Jul 23 21:52:38.294827 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.294754 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeReady" Jul 23 21:52:38.294955 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.294891 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 21:52:38.347748 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.347723 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wzj8m"] Jul 23 21:52:38.349409 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.349394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.353231 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.353210 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 21:52:38.353409 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.353382 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjh2"] Jul 23 21:52:38.353655 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.353638 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 21:52:38.354963 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.354949 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.355314 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.355265 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7bnz\"" Jul 23 21:52:38.359293 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.359272 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 21:52:38.359391 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.359322 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 21:52:38.359391 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.359372 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qfzdd\"" Jul 23 21:52:38.359565 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.359552 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 21:52:38.361692 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.361671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wzj8m"] Jul 23 21:52:38.367000 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.366980 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjh2"] Jul 23 21:52:38.454710 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.454676 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gtmcg"] Jul 23 21:52:38.456512 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.456497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.459468 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.459448 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 21:52:38.459585 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.459452 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 21:52:38.459762 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.459743 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 21:52:38.459855 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.459748 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 21:52:38.459855 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.459793 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpf7h\"" Jul 23 21:52:38.468424 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.468406 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gtmcg"] Jul 23 21:52:38.490943 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.490908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-config-volume\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.491084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.490967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cf054f-80b1-4b3b-8869-43358dc4a75f-cert\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.491084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.490999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-tmp-dir\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.491084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.491032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4krr\" (UniqueName: \"kubernetes.io/projected/11cf054f-80b1-4b3b-8869-43358dc4a75f-kube-api-access-k4krr\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.491084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.491065 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-metrics-tls\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.491296 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.491093 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7r7h\" (UniqueName: \"kubernetes.io/projected/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-kube-api-access-q7r7h\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drm76\" (UniqueName: \"kubernetes.io/projected/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-api-access-drm76\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.592204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-config-volume\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cf054f-80b1-4b3b-8869-43358dc4a75f-cert\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-tmp-dir\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4krr\" (UniqueName: \"kubernetes.io/projected/11cf054f-80b1-4b3b-8869-43358dc4a75f-kube-api-access-k4krr\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-metrics-tls\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7r7h\" (UniqueName: \"kubernetes.io/projected/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-kube-api-access-q7r7h\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f47888e-ef28-45ba-821e-6275fb2d3390-data-volume\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.592538 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f47888e-ef28-45ba-821e-6275fb2d3390-crio-socket\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.592818 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f47888e-ef28-45ba-821e-6275fb2d3390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.592818 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592634 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-tmp-dir\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.592890 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.592871 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-config-volume\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.596238 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.596204 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-metrics-tls\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.596395 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.596374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cf054f-80b1-4b3b-8869-43358dc4a75f-cert\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.602166 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.602146 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7r7h\" (UniqueName: \"kubernetes.io/projected/900f4bc2-eed0-4e0c-bbbd-72759ca182b5-kube-api-access-q7r7h\") pod \"dns-default-wzj8m\" (UID: \"900f4bc2-eed0-4e0c-bbbd-72759ca182b5\") " pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.603726 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.603706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4krr\" (UniqueName: \"kubernetes.io/projected/11cf054f-80b1-4b3b-8869-43358dc4a75f-kube-api-access-k4krr\") pod \"ingress-canary-hjjh2\" (UID: \"11cf054f-80b1-4b3b-8869-43358dc4a75f\") " pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.657696 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.657676 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:38.664962 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.664945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjjh2" Jul 23 21:52:38.693559 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drm76\" (UniqueName: \"kubernetes.io/projected/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-api-access-drm76\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.693705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f47888e-ef28-45ba-821e-6275fb2d3390-data-volume\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.693705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.693705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f47888e-ef28-45ba-821e-6275fb2d3390-crio-socket\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.693705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f47888e-ef28-45ba-821e-6275fb2d3390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.694085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.693969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f47888e-ef28-45ba-821e-6275fb2d3390-data-volume\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.694197 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.694180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f47888e-ef28-45ba-821e-6275fb2d3390-crio-socket\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.694448 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.694405 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.696660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.696428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f47888e-ef28-45ba-821e-6275fb2d3390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.702746 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.702718 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drm76\" (UniqueName: \"kubernetes.io/projected/3f47888e-ef28-45ba-821e-6275fb2d3390-kube-api-access-drm76\") pod \"insights-runtime-extractor-gtmcg\" (UID: \"3f47888e-ef28-45ba-821e-6275fb2d3390\") " pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.765054 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.764789 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gtmcg" Jul 23 21:52:38.771112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.771084 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:52:38.774663 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.773773 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 21:52:38.774663 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.774043 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 21:52:38.775386 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.774952 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2x2\"" Jul 23 21:52:38.804301 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.804244 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wzj8m"] Jul 23 21:52:38.805054 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.805030 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjh2"] Jul 23 21:52:38.808442 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:38.808411 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cf054f_80b1_4b3b_8869_43358dc4a75f.slice/crio-82130f62d3860b471e4c2fd7383f615b85bf25e983f1776049b0c43caf37afcc WatchSource:0}: Error finding container 82130f62d3860b471e4c2fd7383f615b85bf25e983f1776049b0c43caf37afcc: Status 404 returned error can't find the container with id 82130f62d3860b471e4c2fd7383f615b85bf25e983f1776049b0c43caf37afcc Jul 23 21:52:38.809078 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:38.809034 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900f4bc2_eed0_4e0c_bbbd_72759ca182b5.slice/crio-2a4ef08f81ce3ecb868ccb39680e83dc2fb8134f9abf0a3e65e9a8101fc82b5d WatchSource:0}: Error finding container 2a4ef08f81ce3ecb868ccb39680e83dc2fb8134f9abf0a3e65e9a8101fc82b5d: Status 404 returned error can't find the container with id 2a4ef08f81ce3ecb868ccb39680e83dc2fb8134f9abf0a3e65e9a8101fc82b5d Jul 23 21:52:38.889903 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.889871 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gtmcg"] Jul 23 21:52:38.893111 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:38.893079 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f47888e_ef28_45ba_821e_6275fb2d3390.slice/crio-5fa3ce212b117c977dd5e489604ce8eebaa150ab0e8ac586db8bfb180c492417 WatchSource:0}: Error finding container 5fa3ce212b117c977dd5e489604ce8eebaa150ab0e8ac586db8bfb180c492417: Status 404 returned error can't find the container with id 5fa3ce212b117c977dd5e489604ce8eebaa150ab0e8ac586db8bfb180c492417 Jul 23 21:52:38.980835 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.980806 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtmcg" event={"ID":"3f47888e-ef28-45ba-821e-6275fb2d3390","Type":"ContainerStarted","Data":"5fa3ce212b117c977dd5e489604ce8eebaa150ab0e8ac586db8bfb180c492417"} Jul 23 21:52:38.981745 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.981723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjjh2" event={"ID":"11cf054f-80b1-4b3b-8869-43358dc4a75f","Type":"ContainerStarted","Data":"82130f62d3860b471e4c2fd7383f615b85bf25e983f1776049b0c43caf37afcc"} Jul 23 21:52:38.982634 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.982613 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzj8m" event={"ID":"900f4bc2-eed0-4e0c-bbbd-72759ca182b5","Type":"ContainerStarted","Data":"2a4ef08f81ce3ecb868ccb39680e83dc2fb8134f9abf0a3e65e9a8101fc82b5d"} Jul 23 21:52:38.985498 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:38.985469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h5km9" event={"ID":"439b24d5-9bad-4c69-9e1c-4bb7c94631d0","Type":"ContainerStarted","Data":"dda1c9e1eecc866de9fa953afb56e0c4abf0e8d475f6fa62780f720ed4175f09"} Jul 23 21:52:39.009023 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.008984 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h5km9" podStartSLOduration=9.369238028 podStartE2EDuration="38.00896902s" totalImagesPullingTime="28.639730992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:39.008246271 +0000 UTC m=+39.033579805" watchObservedRunningTime="2026-07-23 21:52:39.00896902 +0000 UTC m=+39.034302554" Jul 23 21:52:39.628999 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.628018 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:52:39.631741 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.630726 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.634853 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.634538 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5ff74\"" Jul 23 21:52:39.634853 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.634604 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 21:52:39.634853 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.634538 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 21:52:39.637925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.636391 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 21:52:39.637925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.636659 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 21:52:39.637925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.636895 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 21:52:39.637925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.637105 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 21:52:39.637925 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.637276 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 21:52:39.641046 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.640907 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:52:39.774012 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.772958 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:52:39.774012 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.773351 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:52:39.776765 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.776136 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 21:52:39.776765 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.776364 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 21:52:39.776765 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.776615 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd2p5\"" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp2t\" (UniqueName: \"kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.804784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.804701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.905561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.905400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.905561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.905495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.905561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.905526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp2t\" (UniqueName: \"kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.905812 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.905558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.905812 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.905618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.906103 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.906071 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.906352 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.906327 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.906458 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.906406 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.906752 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.906728 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.911705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.911678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.913745 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.913715 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.925344 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.925260 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp2t\" (UniqueName: \"kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t\") pod \"console-84c9585844-nsqxb\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.946217 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.946047 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:39.986929 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.986904 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zxb92"] Jul 23 21:52:39.991191 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.991166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:39.994118 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.993917 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 23 21:52:39.994118 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.994022 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vnzph\"" Jul 23 21:52:39.995827 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.994507 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 21:52:39.995827 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.994523 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 23 21:52:39.995976 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.995908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 21:52:39.996308 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.996135 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 21:52:40.000036 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:39.999975 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zxb92"] Jul 23 21:52:40.108048 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.108017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.108210 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.108105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.108210 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.108156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90109c89-0fc2-4084-b4c0-21f192e763cd-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.108331 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.108250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcqc\" (UniqueName: \"kubernetes.io/projected/90109c89-0fc2-4084-b4c0-21f192e763cd-kube-api-access-rqcqc\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.121946 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.121553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:52:40.209437 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.209362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.209437 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.209414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90109c89-0fc2-4084-b4c0-21f192e763cd-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.209657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.209507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcqc\" (UniqueName: \"kubernetes.io/projected/90109c89-0fc2-4084-b4c0-21f192e763cd-kube-api-access-rqcqc\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.209657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.209599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.210204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.210144 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90109c89-0fc2-4084-b4c0-21f192e763cd-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.214557 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.214476 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.214878 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.214836 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90109c89-0fc2-4084-b4c0-21f192e763cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.221025 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.221002 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcqc\" (UniqueName: \"kubernetes.io/projected/90109c89-0fc2-4084-b4c0-21f192e763cd-kube-api-access-rqcqc\") pod \"prometheus-operator-5945b5cfd-zxb92\" (UID: \"90109c89-0fc2-4084-b4c0-21f192e763cd\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.307974 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.307937 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" Jul 23 21:52:40.690429 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.690341 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:52:40.707667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.707638 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.710001 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.709936 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:52:40.718605 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.718581 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q2z\" (UniqueName: \"kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814699 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814759 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.814991 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.814930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916214 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.915983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q2z\" (UniqueName: \"kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916570 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916570 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.916570 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.916458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.917409 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.917381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.917535 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.917428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.917701 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.917676 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.917876 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.917852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.919823 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.919798 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.920195 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.920159 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.926427 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.926384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q2z\" (UniqueName: \"kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z\") pod \"console-6f5df5474b-r8qfh\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:40.996573 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:40.996529 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9585844-nsqxb" event={"ID":"f656b369-2dea-4b32-96f5-2abbfd6d74c5","Type":"ContainerStarted","Data":"4d0bac4f330e80fb3cd513c3064cb723272620fc07d506d68930fd5ba5da7cfe"} Jul 23 21:52:41.020447 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:41.020420 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:41.049300 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:41.049201 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zxb92"] Jul 23 21:52:41.052122 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:41.052090 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90109c89_0fc2_4084_b4c0_21f192e763cd.slice/crio-26b318682a152f954864049a419a4830a468b171cd5eb41c0b89ebfe726747b0 WatchSource:0}: Error finding container 26b318682a152f954864049a419a4830a468b171cd5eb41c0b89ebfe726747b0: Status 404 returned error can't find the container with id 26b318682a152f954864049a419a4830a468b171cd5eb41c0b89ebfe726747b0 Jul 23 21:52:41.161296 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:41.161210 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:52:42.000670 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:42.000605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" event={"ID":"90109c89-0fc2-4084-b4c0-21f192e763cd","Type":"ContainerStarted","Data":"26b318682a152f954864049a419a4830a468b171cd5eb41c0b89ebfe726747b0"} Jul 23 21:52:42.139310 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:42.139269 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8ffb77_d3bb_48f0_93c8_a2b7cae67da9.slice/crio-1a43ef2ee2a36419fada79a93a18f5b77a8cdff835fa3fd6afc57cc205182576 WatchSource:0}: Error finding container 1a43ef2ee2a36419fada79a93a18f5b77a8cdff835fa3fd6afc57cc205182576: Status 404 returned error can't find the container with id 1a43ef2ee2a36419fada79a93a18f5b77a8cdff835fa3fd6afc57cc205182576 Jul 23 21:52:43.005256 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:43.005220 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5df5474b-r8qfh" event={"ID":"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9","Type":"ContainerStarted","Data":"1a43ef2ee2a36419fada79a93a18f5b77a8cdff835fa3fd6afc57cc205182576"} Jul 23 21:52:49.025562 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.025298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9585844-nsqxb" event={"ID":"f656b369-2dea-4b32-96f5-2abbfd6d74c5","Type":"ContainerStarted","Data":"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716"} Jul 23 21:52:49.027863 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.027833 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" event={"ID":"90109c89-0fc2-4084-b4c0-21f192e763cd","Type":"ContainerStarted","Data":"d2813d4f3f82d88d8271d2982fcee272495fb1c23b9533b48f533c0d52aaed2d"} Jul 23 21:52:49.027980 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.027883 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" event={"ID":"90109c89-0fc2-4084-b4c0-21f192e763cd","Type":"ContainerStarted","Data":"7d8a6192b6c1e5610f6dbcc48cfa707270281b5a8d8caf4ad361eaac56d9c88d"} Jul 23 21:52:49.030384 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.030362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5df5474b-r8qfh" event={"ID":"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9","Type":"ContainerStarted","Data":"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a"} Jul 23 21:52:49.032773 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.032732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzj8m" event={"ID":"900f4bc2-eed0-4e0c-bbbd-72759ca182b5","Type":"ContainerStarted","Data":"1e04bbee84b07b0519ace9db95d26ca70928291a020bf37151f911dfe3cfbc76"} Jul 23 21:52:49.032866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.032799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzj8m" event={"ID":"900f4bc2-eed0-4e0c-bbbd-72759ca182b5","Type":"ContainerStarted","Data":"fc3f2f12f93d81117695ea4b70fbeea4479d044acbb9df944df1dde4c76ec809"} Jul 23 21:52:49.032959 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.032944 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wzj8m" Jul 23 21:52:49.034637 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.034612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtmcg" event={"ID":"3f47888e-ef28-45ba-821e-6275fb2d3390","Type":"ContainerStarted","Data":"57ae3edf1b4b4c569cb1654489eebee778169588b42fb7544313081c05a5507a"} Jul 23 21:52:49.036508 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.036464 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjjh2" event={"ID":"11cf054f-80b1-4b3b-8869-43358dc4a75f","Type":"ContainerStarted","Data":"1597b6a795df4e5430c6ab2ad3ca652d6324c150bd1d57e3837776e6d72281d6"} Jul 23 21:52:49.046220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.046178 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c9585844-nsqxb" podStartSLOduration=2.182255815 podStartE2EDuration="10.046166641s" totalImagesPullingTime="7.863910826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:49.045830354 +0000 UTC m=+49.071163890" watchObservedRunningTime="2026-07-23 21:52:49.046166641 +0000 UTC m=+49.071500236" Jul 23 21:52:49.064823 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.064763 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wzj8m" podStartSLOduration=4.67997668 podStartE2EDuration="11.064744179s" totalImagesPullingTime="6.384767499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:49.063296999 +0000 UTC m=+49.088630686" watchObservedRunningTime="2026-07-23 21:52:49.064744179 +0000 UTC m=+49.090077714" Jul 23 21:52:49.082438 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.082386 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5df5474b-r8qfh" podStartSLOduration=3.0254454 podStartE2EDuration="9.082369768s" totalImagesPullingTime="6.056924368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:49.081178985 +0000 UTC m=+49.106512511" watchObservedRunningTime="2026-07-23 21:52:49.082369768 +0000 UTC m=+49.107703285" Jul 23 21:52:49.098150 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.098104 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjjh2" podStartSLOduration=4.267060278 podStartE2EDuration="11.098090582s" totalImagesPullingTime="6.831030304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:49.097253126 +0000 UTC m=+49.122586660" watchObservedRunningTime="2026-07-23 21:52:49.098090582 +0000 UTC m=+49.123424116" Jul 23 21:52:49.113623 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.113566 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zxb92" podStartSLOduration=3.16272488 podStartE2EDuration="10.113545508s" totalImagesPullingTime="6.950820628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:49.113457645 +0000 UTC m=+49.138791180" watchObservedRunningTime="2026-07-23 21:52:49.113545508 +0000 UTC m=+49.138879044" Jul 23 21:52:49.946327 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.946294 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:49.946530 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.946348 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:49.951895 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:49.951871 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:50.043811 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:50.043772 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:52:51.020756 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.020725 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:51.020899 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.020816 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:51.025459 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.025433 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:51.047778 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.047751 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:52:51.172875 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.172841 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:52:51.359725 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.359649 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6"] Jul 23 21:52:51.389705 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.389672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6"] Jul 23 21:52:51.389882 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.389737 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.392503 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.392462 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 21:52:51.393166 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.393135 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m2qbv\"" Jul 23 21:52:51.393613 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.393592 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 21:52:51.457684 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.457650 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m9tj6"] Jul 23 21:52:51.489033 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.489005 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xt4lv"] Jul 23 21:52:51.489228 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.489153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.490422 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.490399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.490544 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.490468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25aecb24-e466-4359-9884-c887b54474e3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.490544 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.490529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.490638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.490565 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84qn2\" (UniqueName: \"kubernetes.io/projected/25aecb24-e466-4359-9884-c887b54474e3-kube-api-access-84qn2\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.491756 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.491725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 21:52:51.491960 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.491942 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 21:52:51.492107 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.492076 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5hwvl\"" Jul 23 21:52:51.492201 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.492172 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 21:52:51.510945 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.510807 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m9tj6"] Jul 23 21:52:51.510945 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.510930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.513208 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.513180 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 21:52:51.513208 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.513188 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gzhmv\"" Jul 23 21:52:51.513393 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.513327 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 21:52:51.513393 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.513345 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 21:52:51.591833 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-root\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25aecb24-e466-4359-9884-c887b54474e3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-textfile\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.591956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84qn2\" (UniqueName: \"kubernetes.io/projected/25aecb24-e466-4359-9884-c887b54474e3-kube-api-access-84qn2\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqgh\" (UniqueName: \"kubernetes.io/projected/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-api-access-rjqgh\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-tls\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-metrics-client-ca\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-sys\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7djs\" (UniqueName: \"kubernetes.io/projected/086846b1-3bf5-4845-af98-154e46dd4fac-kube-api-access-n7djs\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-accelerators-collector-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.592563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-wtmp\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.592759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.592688 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25aecb24-e466-4359-9884-c887b54474e3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.594943 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.594919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.595053 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.595020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25aecb24-e466-4359-9884-c887b54474e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.603786 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.603758 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84qn2\" (UniqueName: \"kubernetes.io/projected/25aecb24-e466-4359-9884-c887b54474e3-kube-api-access-84qn2\") pod \"openshift-state-metrics-79f6958cdf-stxw6\" (UID: \"25aecb24-e466-4359-9884-c887b54474e3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.693615 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-tls\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693615 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-metrics-client-ca\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693791 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.693791 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-sys\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693791 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7djs\" (UniqueName: \"kubernetes.io/projected/086846b1-3bf5-4845-af98-154e46dd4fac-kube-api-access-n7djs\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693782 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-sys\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-accelerators-collector-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.693906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694025 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-wtmp\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694025 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.693984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694120 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-root\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694120 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694218 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-wtmp\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694218 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694168 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/086846b1-3bf5-4845-af98-154e46dd4fac-root\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694314 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694272 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694314 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-textfile\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694411 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694411 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694250 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-metrics-client-ca\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694411 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694571 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqgh\" (UniqueName: \"kubernetes.io/projected/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-api-access-rjqgh\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694571 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694535 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-accelerators-collector-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.694667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694592 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-textfile\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.694927 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.694901 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.696563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.696535 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-tls\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.696657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.696570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/086846b1-3bf5-4845-af98-154e46dd4fac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.696657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.696631 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.696657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.696638 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.702883 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.701250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" Jul 23 21:52:51.705260 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.703528 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7djs\" (UniqueName: \"kubernetes.io/projected/086846b1-3bf5-4845-af98-154e46dd4fac-kube-api-access-n7djs\") pod \"node-exporter-xt4lv\" (UID: \"086846b1-3bf5-4845-af98-154e46dd4fac\") " pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.705260 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.703869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqgh\" (UniqueName: \"kubernetes.io/projected/75ab352b-901d-4f9b-9d4e-e413a4a0d0ae-kube-api-access-rjqgh\") pod \"kube-state-metrics-56c5594674-m9tj6\" (UID: \"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.798768 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.798739 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" Jul 23 21:52:51.820969 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:51.820945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xt4lv" Jul 23 21:52:51.888756 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:51.888670 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086846b1_3bf5_4845_af98_154e46dd4fac.slice/crio-7d1ed6ab1cdad2da353553b88baadc6cbc3571c2d213c0178170cf4cbe663cd7 WatchSource:0}: Error finding container 7d1ed6ab1cdad2da353553b88baadc6cbc3571c2d213c0178170cf4cbe663cd7: Status 404 returned error can't find the container with id 7d1ed6ab1cdad2da353553b88baadc6cbc3571c2d213c0178170cf4cbe663cd7 Jul 23 21:52:52.009615 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.009591 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6"] Jul 23 21:52:52.011413 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:52.011384 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aecb24_e466_4359_9884_c887b54474e3.slice/crio-d4a590074b8fe87a28dbdb20597d274d5ec1291365b8eedfc362513da1cc7664 WatchSource:0}: Error finding container d4a590074b8fe87a28dbdb20597d274d5ec1291365b8eedfc362513da1cc7664: Status 404 returned error can't find the container with id d4a590074b8fe87a28dbdb20597d274d5ec1291365b8eedfc362513da1cc7664 Jul 23 21:52:52.024966 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.024943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m9tj6"] Jul 23 21:52:52.027904 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:52.027871 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ab352b_901d_4f9b_9d4e_e413a4a0d0ae.slice/crio-661f668b423df366e69895df86c296f6a5674d501dc8ee8f086a741889197f8d WatchSource:0}: Error finding container 661f668b423df366e69895df86c296f6a5674d501dc8ee8f086a741889197f8d: Status 404 returned error can't find the container with id 661f668b423df366e69895df86c296f6a5674d501dc8ee8f086a741889197f8d Jul 23 21:52:52.047191 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.047167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xt4lv" event={"ID":"086846b1-3bf5-4845-af98-154e46dd4fac","Type":"ContainerStarted","Data":"7d1ed6ab1cdad2da353553b88baadc6cbc3571c2d213c0178170cf4cbe663cd7"} Jul 23 21:52:52.048241 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.048206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" event={"ID":"25aecb24-e466-4359-9884-c887b54474e3","Type":"ContainerStarted","Data":"d4a590074b8fe87a28dbdb20597d274d5ec1291365b8eedfc362513da1cc7664"} Jul 23 21:52:52.049414 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.049391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" event={"ID":"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae","Type":"ContainerStarted","Data":"661f668b423df366e69895df86c296f6a5674d501dc8ee8f086a741889197f8d"} Jul 23 21:52:52.419746 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.419706 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:52:52.445916 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.445888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:52:52.446072 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.446045 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.448621 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.448598 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 21:52:52.448885 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.448835 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 21:52:52.449019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449005 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449554 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449611 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449692 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449756 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m9frt\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449852 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 21:52:52.449918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.449868 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 21:52:52.500565 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.500723 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr78\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.500723 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.500723 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.500723 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500759 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.500923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.501031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.501204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.501096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602563 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr78\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.602709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602692 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.602940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.603385 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:52.603144 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle podName:af3d37d7-331f-4c97-8f52-037c32e983b9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:53.103123878 +0000 UTC m=+53.128457405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9") : configmap references non-existent config key: ca-bundle.crt Jul 23 21:52:52.603385 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.603321 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.604018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.603989 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.605330 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.605305 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.605439 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:52.605417 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 23 21:52:52.605521 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:52:52.605466 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls podName:af3d37d7-331f-4c97-8f52-037c32e983b9 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:53.10544969 +0000 UTC m=+53.130783220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9") : secret "alertmanager-main-tls" not found Jul 23 21:52:52.606080 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.606017 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.606179 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.606104 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.606569 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.606523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.606569 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.606531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.607446 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.607415 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.607973 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.607957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.608572 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.608547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:52.611325 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:52.611303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr78\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.054091 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.054048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtmcg" event={"ID":"3f47888e-ef28-45ba-821e-6275fb2d3390","Type":"ContainerStarted","Data":"83306e709051e9a52a2934c977e16bb6c44d39caceda2d9670ceace9d8ffc8c9"} Jul 23 21:52:53.055472 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.055440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" event={"ID":"25aecb24-e466-4359-9884-c887b54474e3","Type":"ContainerStarted","Data":"0cdcb7a6394fa0177004133995587d54449e1a9412aa4737a645f202b8a8a578"} Jul 23 21:52:53.055599 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.055501 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" event={"ID":"25aecb24-e466-4359-9884-c887b54474e3","Type":"ContainerStarted","Data":"232860238b5ebaa189fa7f136eda7f22de211cf1b21fa09084542b707da313f0"} Jul 23 21:52:53.108312 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.108283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.108596 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.108572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.109167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.109141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.110808 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.110786 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.357598 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.357316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:52:53.435503 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.435198 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6887688fc8-7rt52"] Jul 23 21:52:53.448573 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.448539 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.451706 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.451665 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6887688fc8-7rt52"] Jul 23 21:52:53.454167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.453948 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 21:52:53.454167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.453982 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f0eoeh3c77a6m\"" Jul 23 21:52:53.454167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.454082 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 21:52:53.454167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.454133 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 21:52:53.454608 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.454563 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r578c\"" Jul 23 21:52:53.454871 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.454664 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 21:52:53.456101 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.456054 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwr7\" (UniqueName: \"kubernetes.io/projected/a38c4995-e680-4da3-b8b3-7f528b49d86a-kube-api-access-xxwr7\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-grpc-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a38c4995-e680-4da3-b8b3-7f528b49d86a-metrics-client-ca\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.512912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.512802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613564 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-grpc-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613564 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a38c4995-e680-4da3-b8b3-7f528b49d86a-metrics-client-ca\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613646 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.613977 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.613948 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.614051 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.614027 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.614112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.614097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwr7\" (UniqueName: \"kubernetes.io/projected/a38c4995-e680-4da3-b8b3-7f528b49d86a-kube-api-access-xxwr7\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.614850 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.614803 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a38c4995-e680-4da3-b8b3-7f528b49d86a-metrics-client-ca\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.617442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.617416 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.619806 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.619764 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.619967 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.619923 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-grpc-tls\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.620070 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.619969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.620174 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.620154 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.620590 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.620547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a38c4995-e680-4da3-b8b3-7f528b49d86a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.625525 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.625469 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwr7\" (UniqueName: \"kubernetes.io/projected/a38c4995-e680-4da3-b8b3-7f528b49d86a-kube-api-access-xxwr7\") pod \"thanos-querier-6887688fc8-7rt52\" (UID: \"a38c4995-e680-4da3-b8b3-7f528b49d86a\") " pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.762983 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.762957 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:52:53.902185 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:53.902155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:52:54.275842 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:54.275810 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3d37d7_331f_4c97_8f52_037c32e983b9.slice/crio-028493eacf6cfbd1d6757509246784e03953bfad2f6d8023175672ebfa63ea93 WatchSource:0}: Error finding container 028493eacf6cfbd1d6757509246784e03953bfad2f6d8023175672ebfa63ea93: Status 404 returned error can't find the container with id 028493eacf6cfbd1d6757509246784e03953bfad2f6d8023175672ebfa63ea93 Jul 23 21:52:54.633239 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:54.633165 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6887688fc8-7rt52"] Jul 23 21:52:55.066224 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.066122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" event={"ID":"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae","Type":"ContainerStarted","Data":"1c49e7ca2bda5c2a3cc862d15ed51a6d5199d4cb4e320752fd3beddc86cb02eb"} Jul 23 21:52:55.066394 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.066235 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" event={"ID":"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae","Type":"ContainerStarted","Data":"cfd2aeb1cc4ed1205ffbcbbc26253b50426574efae5ab4a13645d403abd7a7cf"} Jul 23 21:52:55.066394 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.066258 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" event={"ID":"75ab352b-901d-4f9b-9d4e-e413a4a0d0ae","Type":"ContainerStarted","Data":"dd449ace3d82557eef242715944c21a3902115be8eee4b5ffceaeb39a3092045"} Jul 23 21:52:55.067610 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.067579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"4b96fa91a3a4358ea0bf49863beef55a44fb182a00ceb375295d835b82f89349"} Jul 23 21:52:55.069313 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.069286 2569 generic.go:356] "Generic (PLEG): container finished" podID="086846b1-3bf5-4845-af98-154e46dd4fac" containerID="7d0fb21320eb2c3202109688ca57d145aeecbee988067091b8badf86e57f5dee" exitCode=0 Jul 23 21:52:55.069438 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.069356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xt4lv" event={"ID":"086846b1-3bf5-4845-af98-154e46dd4fac","Type":"ContainerDied","Data":"7d0fb21320eb2c3202109688ca57d145aeecbee988067091b8badf86e57f5dee"} Jul 23 21:52:55.070852 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.070831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"028493eacf6cfbd1d6757509246784e03953bfad2f6d8023175672ebfa63ea93"} Jul 23 21:52:55.073047 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.073026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" event={"ID":"25aecb24-e466-4359-9884-c887b54474e3","Type":"ContainerStarted","Data":"018af63977b591417338a5804a81b2fb1a2bdc45c23f646e246903ea2061158e"} Jul 23 21:52:55.086598 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.086546 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-m9tj6" podStartSLOduration=1.633698037 podStartE2EDuration="4.086528476s" totalImagesPullingTime="2.452830439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:55.085559423 +0000 UTC m=+55.110892957" watchObservedRunningTime="2026-07-23 21:52:55.086528476 +0000 UTC m=+55.111862010" Jul 23 21:52:55.127997 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.127940 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-stxw6" podStartSLOduration=2.036591143 podStartE2EDuration="4.127922633s" totalImagesPullingTime="2.09133149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:55.12736216 +0000 UTC m=+55.152695696" watchObservedRunningTime="2026-07-23 21:52:55.127922633 +0000 UTC m=+55.153256168" Jul 23 21:52:55.855430 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.855395 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl"] Jul 23 21:52:55.999756 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.999720 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl"] Jul 23 21:52:55.999910 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:55.999853 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.004312 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.004288 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 21:52:56.004448 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.004401 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 21:52:56.005597 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.005573 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4f75f\"" Jul 23 21:52:56.005720 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.005597 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aknc0nhdgrcff\"" Jul 23 21:52:56.005720 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.005606 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 21:52:56.006038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.006018 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 21:52:56.035980 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.035951 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/94d9f069-4bf8-4d3e-a534-7e48762f50f2-audit-log\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036110 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.035985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-metrics-server-audit-profiles\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036110 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.036018 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-client-certs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036110 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.036076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzhs\" (UniqueName: \"kubernetes.io/projected/94d9f069-4bf8-4d3e-a534-7e48762f50f2-kube-api-access-wpzhs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036222 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.036170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-client-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036222 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.036214 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-tls\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.036303 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.036248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137135 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137316 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/94d9f069-4bf8-4d3e-a534-7e48762f50f2-audit-log\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137455 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-metrics-server-audit-profiles\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137550 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-client-certs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137618 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzhs\" (UniqueName: \"kubernetes.io/projected/94d9f069-4bf8-4d3e-a534-7e48762f50f2-kube-api-access-wpzhs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137673 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137639 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/94d9f069-4bf8-4d3e-a534-7e48762f50f2-audit-log\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137726 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-client-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.137779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137731 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.138234 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.137977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-tls\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.138692 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.138637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/94d9f069-4bf8-4d3e-a534-7e48762f50f2-metrics-server-audit-profiles\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.140766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.140723 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-tls\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.140766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.140732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-secret-metrics-server-client-certs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.140907 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.140782 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/94d9f069-4bf8-4d3e-a534-7e48762f50f2-client-ca-bundle\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.152089 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.152057 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:52:56.157109 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.157087 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzhs\" (UniqueName: \"kubernetes.io/projected/94d9f069-4bf8-4d3e-a534-7e48762f50f2-kube-api-access-wpzhs\") pod \"metrics-server-5cc8ffdcb8-k42xl\" (UID: \"94d9f069-4bf8-4d3e-a534-7e48762f50f2\") " pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.228557 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.228519 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm"] Jul 23 21:52:56.228741 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.228689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.261764 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.261740 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:52:56.261889 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.261778 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm"] Jul 23 21:52:56.261889 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.261870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.268405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.268376 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 23 21:52:56.268546 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.268427 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kjz5w\"" Jul 23 21:52:56.310030 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.309997 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:52:56.339496 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339645 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339645 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gjg\" (UniqueName: \"kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339645 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339645 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34bbc941-25c2-4878-8709-249385e4adec-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.339839 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339839 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339839 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339766 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.339839 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.339799 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnn6d\" (UniqueName: \"kubernetes.io/projected/34bbc941-25c2-4878-8709-249385e4adec-kube-api-access-gnn6d\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.440289 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440446 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gjg\" (UniqueName: \"kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440446 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440446 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34bbc941-25c2-4878-8709-249385e4adec-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.440638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.440638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnn6d\" (UniqueName: \"kubernetes.io/projected/34bbc941-25c2-4878-8709-249385e4adec-kube-api-access-gnn6d\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.440822 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.440638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.441103 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.441073 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.441197 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.441129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.441431 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.441408 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.441561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.441547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.443037 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.443016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.443202 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.443184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34bbc941-25c2-4878-8709-249385e4adec-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.443496 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.443438 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.449639 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.449612 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnn6d\" (UniqueName: \"kubernetes.io/projected/34bbc941-25c2-4878-8709-249385e4adec-kube-api-access-gnn6d\") pod \"monitoring-plugin-7b7b7bc8bf-tnxwm\" (UID: \"34bbc941-25c2-4878-8709-249385e4adec\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.450251 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.450233 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gjg\" (UniqueName: \"kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg\") pod \"console-5d58c55694-x2sp2\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.542076 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.541741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:52:56.572000 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.571832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:52:56.622785 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.622740 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6584c8969f-f6lc6"] Jul 23 21:52:56.625473 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.625456 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.630616 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.630575 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl"] Jul 23 21:52:56.630796 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.630780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 21:52:56.631556 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.631436 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 21:52:56.631664 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.631628 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rrt9r\"" Jul 23 21:52:56.631751 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.631709 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 21:52:56.631843 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.631821 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 21:52:56.631843 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.631821 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 21:52:56.640543 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.640345 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 21:52:56.649756 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.647472 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6584c8969f-f6lc6"] Jul 23 21:52:56.712390 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.712350 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:52:56.714870 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:56.714838 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ba28d1_fcce_46af_809e_4be124bf576c.slice/crio-fc0f1380429fec0c098328fd613aadf40c443261614d7676f00d544770488bc4 WatchSource:0}: Error finding container fc0f1380429fec0c098328fd613aadf40c443261614d7676f00d544770488bc4: Status 404 returned error can't find the container with id fc0f1380429fec0c098328fd613aadf40c443261614d7676f00d544770488bc4 Jul 23 21:52:56.736903 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.736871 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm"] Jul 23 21:52:56.740307 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:56.740281 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bbc941_25c2_4878_8709_249385e4adec.slice/crio-17b072e354398c330b9674e803651fde8dcbd3722aad6826ad7d6474107afdab WatchSource:0}: Error finding container 17b072e354398c330b9674e803651fde8dcbd3722aad6826ad7d6474107afdab: Status 404 returned error can't find the container with id 17b072e354398c330b9674e803651fde8dcbd3722aad6826ad7d6474107afdab Jul 23 21:52:56.743779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.743658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.743779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.743706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.743779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.743743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-metrics-client-ca\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.743959 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.743872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-federate-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.744018 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.743981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dt7\" (UniqueName: \"kubernetes.io/projected/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-kube-api-access-p7dt7\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.744074 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.744012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.744074 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.744036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-serving-certs-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.744168 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.744095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.844867 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.844835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-serving-certs-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845032 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845103 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845103 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-metrics-client-ca\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-federate-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dt7\" (UniqueName: \"kubernetes.io/projected/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-kube-api-access-p7dt7\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845361 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.845501 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.845442 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-serving-certs-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.846180 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.846066 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-metrics-client-ca\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.847236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.847191 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.847985 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.847949 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.848388 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.848357 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-telemeter-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.848726 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.848709 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.849246 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.849202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-federate-client-tls\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.857118 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.857094 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dt7\" (UniqueName: \"kubernetes.io/projected/16f4ee53-5b4d-4c6c-b497-02d5ba76ee88-kube-api-access-p7dt7\") pod \"telemeter-client-6584c8969f-f6lc6\" (UID: \"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88\") " pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:56.957648 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:56.957610 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" Jul 23 21:52:57.085027 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.084992 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d58c55694-x2sp2" event={"ID":"28ba28d1-fcce-46af-809e-4be124bf576c","Type":"ContainerStarted","Data":"687df32de931ae60a9784c6f8fec1bc396fa246c43751ba9a71411a61d8d8837"} Jul 23 21:52:57.085195 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.085064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d58c55694-x2sp2" event={"ID":"28ba28d1-fcce-46af-809e-4be124bf576c","Type":"ContainerStarted","Data":"fc0f1380429fec0c098328fd613aadf40c443261614d7676f00d544770488bc4"} Jul 23 21:52:57.087768 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.087630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gtmcg" event={"ID":"3f47888e-ef28-45ba-821e-6275fb2d3390","Type":"ContainerStarted","Data":"3d8b5db7bea9855dfff064333782385f04fc7dca36b94e1de72b397fde751159"} Jul 23 21:52:57.091562 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.091475 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" event={"ID":"34bbc941-25c2-4878-8709-249385e4adec","Type":"ContainerStarted","Data":"17b072e354398c330b9674e803651fde8dcbd3722aad6826ad7d6474107afdab"} Jul 23 21:52:57.093311 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.093285 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" event={"ID":"94d9f069-4bf8-4d3e-a534-7e48762f50f2","Type":"ContainerStarted","Data":"80d9f3bd41c24dfbadf1c017552eec0634470218907f7e2576698b0c01379c21"} Jul 23 21:52:57.096383 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.096359 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xt4lv" event={"ID":"086846b1-3bf5-4845-af98-154e46dd4fac","Type":"ContainerStarted","Data":"05f5246748b9b861dc9ca5f34ab7a3f9c78938ab0dc8a30e00df7bd385df8006"} Jul 23 21:52:57.096475 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.096406 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xt4lv" event={"ID":"086846b1-3bf5-4845-af98-154e46dd4fac","Type":"ContainerStarted","Data":"d9f3f7bb751461243af9e096804f5873640a78ec303d45689e59efa113bc1a7c"} Jul 23 21:52:57.106525 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.106460 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d58c55694-x2sp2" podStartSLOduration=1.106443582 podStartE2EDuration="1.106443582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:57.105808397 +0000 UTC m=+57.131141934" watchObservedRunningTime="2026-07-23 21:52:57.106443582 +0000 UTC m=+57.131777119" Jul 23 21:52:57.118926 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.118858 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6584c8969f-f6lc6"] Jul 23 21:52:57.128092 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.128053 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gtmcg" podStartSLOduration=1.792781078 podStartE2EDuration="19.128038759s" totalImagesPullingTime="17.335257681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:57.125769819 +0000 UTC m=+57.151103355" watchObservedRunningTime="2026-07-23 21:52:57.128038759 +0000 UTC m=+57.153372272" Jul 23 21:52:57.151693 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.151596 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xt4lv" podStartSLOduration=4.154117177 podStartE2EDuration="6.151577616s" totalImagesPullingTime="1.856570276s" totalInitContainerRuntime="140.890163ms" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:57.149957255 +0000 UTC m=+57.175290790" watchObservedRunningTime="2026-07-23 21:52:57.151577616 +0000 UTC m=+57.176911152" Jul 23 21:52:57.287116 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:52:57.287027 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f4ee53_5b4d_4c6c_b497_02d5ba76ee88.slice/crio-b02c49de3d3aa0281efc715a634898ff75c1fa767686e86a810c733ba2e902fe WatchSource:0}: Error finding container b02c49de3d3aa0281efc715a634898ff75c1fa767686e86a810c733ba2e902fe: Status 404 returned error can't find the container with id b02c49de3d3aa0281efc715a634898ff75c1fa767686e86a810c733ba2e902fe Jul 23 21:52:57.831510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.829922 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:52:57.835607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.832910 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.842817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.841895 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 21:52:57.842817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.842204 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 21:52:57.842817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.842405 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 21:52:57.842817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.842552 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 21:52:57.842817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.842724 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 21:52:57.843512 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843190 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 21:52:57.843512 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843251 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 21:52:57.843673 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843633 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 21:52:57.844912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843747 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 21:52:57.844912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843868 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j9zpm\"" Jul 23 21:52:57.844912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.843987 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 21:52:57.844912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.844148 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 21:52:57.844912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.844277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-36ire8h9kjgum\"" Jul 23 21:52:57.846567 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.846345 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 21:52:57.851941 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.851677 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.854925 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.855607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855597 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856252 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.855623 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856366 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856467 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856611 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856708 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856803 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.856909 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857003 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.856990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857201 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857626 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857743 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857851 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.857945 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.857931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.858053 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.858037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.859265 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.859047 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 21:52:57.959725 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.959898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.959898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.959898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.959898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.959898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.959977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960149 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.960527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.960322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.963731 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.963440 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.964736 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.964117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.965096 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.964908 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.968562 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.967667 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.968562 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.967987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.968562 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.968472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.969212 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.969187 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.970553 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.970050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.971265 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.970820 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.971363 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.971298 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.971820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.971776 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.971919 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.971847 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.972519 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.972460 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.972618 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.972527 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.973452 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.973083 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.973452 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.973411 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.973619 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.973508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.973619 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.973555 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.974820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.974763 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:57.974898 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:57.974858 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:58.105167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:58.105038 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" event={"ID":"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88","Type":"ContainerStarted","Data":"b02c49de3d3aa0281efc715a634898ff75c1fa767686e86a810c733ba2e902fe"} Jul 23 21:52:58.107255 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:58.107223 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="505c6504ea6f80a2acd67c7b626525e7ba59db5ced9c9fddbf25720cea10c74d" exitCode=0 Jul 23 21:52:58.110302 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:58.108498 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"505c6504ea6f80a2acd67c7b626525e7ba59db5ced9c9fddbf25720cea10c74d"} Jul 23 21:52:58.148891 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:58.148796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:52:59.042215 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:52:59.042167 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wzj8m" Jul 23 21:53:00.253230 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:00.253108 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:01.123022 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.122980 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" event={"ID":"34bbc941-25c2-4878-8709-249385e4adec","Type":"ContainerStarted","Data":"a3449fb75845abdd961e810cf87590f6815cc8698400bf19f91397e190ee77eb"} Jul 23 21:53:01.123284 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.123263 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:53:01.126071 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.125430 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" event={"ID":"94d9f069-4bf8-4d3e-a534-7e48762f50f2","Type":"ContainerStarted","Data":"dc7a673fc3e941c8f671683943c848ae59f3525ebe512dcdb5e5558cd75000f9"} Jul 23 21:53:01.129104 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.129074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" event={"ID":"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88","Type":"ContainerStarted","Data":"b4b6425f565bdf905434088d26b515928d3932411ce6da9297dc6839a175b24c"} Jul 23 21:53:01.129200 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.129125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" event={"ID":"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88","Type":"ContainerStarted","Data":"95caaf6e51a1e8ecceafb8ed1ef1ab5e5bfa6a55b9b73dabddf0400a7854f022"} Jul 23 21:53:01.129200 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.129155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" event={"ID":"16f4ee53-5b4d-4c6c-b497-02d5ba76ee88","Type":"ContainerStarted","Data":"867573f49b0181eb2811deba47cec34f95ca6b5e709660ac1013c038871c1930"} Jul 23 21:53:01.130463 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.130443 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" Jul 23 21:53:01.131297 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.131272 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" exitCode=0 Jul 23 21:53:01.131382 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.131349 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} Jul 23 21:53:01.131435 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.131419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"4eb53867e15102a37c40c02219aa8afaddbcdb8cb04423f27ed9788a1b0ae632"} Jul 23 21:53:01.135715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.135446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"6fb40a030cf80c505d950f0b34295be2359317f9e137ba434df274a434f5e091"} Jul 23 21:53:01.135715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.135539 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"3cd7a3377fd2fa95a694e9e70b2676261dcad437002dbf9af5778422c25619c3"} Jul 23 21:53:01.135715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.135587 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"93244bd0df218bb501807f318ceb8edf52629095931daac18730a30bac094779"} Jul 23 21:53:01.135715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.135623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"950b59fdb7135c688308c7aca4b17299cae92e2885148ea94aeef0b7e0091696"} Jul 23 21:53:01.135715 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.135664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"adfb9bc8fb2ef9ec690034e714cbb58791e711ae3b5e3444b0134cc2c1238429"} Jul 23 21:53:01.138213 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.138088 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"df512a63c36b8f6b16b456d2d0a3ddc15e31193a875f3bf381edd3a69fea3f50"} Jul 23 21:53:01.138213 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.138153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"e9d9a517399dcc0b11dad41857b5c02f9ca6a2ff2f569cf3e2e21f900e73f035"} Jul 23 21:53:01.138213 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.138194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"5731778373bd868898993594f2cd63c629644b588be0b615f98eabe4265b1bac"} Jul 23 21:53:01.142759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.142711 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tnxwm" podStartSLOduration=1.793375189 podStartE2EDuration="5.142696182s" totalImagesPullingTime="3.349320993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:01.140792106 +0000 UTC m=+61.166125664" watchObservedRunningTime="2026-07-23 21:53:01.142696182 +0000 UTC m=+61.168029718" Jul 23 21:53:01.175147 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.175085 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6584c8969f-f6lc6" podStartSLOduration=2.372340195 podStartE2EDuration="5.175052223s" totalImagesPullingTime="2.802712028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:01.174284107 +0000 UTC m=+61.199617643" watchObservedRunningTime="2026-07-23 21:53:01.175052223 +0000 UTC m=+61.200385760" Jul 23 21:53:01.969866 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:01.969538 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf658" Jul 23 21:53:02.000985 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.000921 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" podStartSLOduration=3.549442814 podStartE2EDuration="7.000897587s" totalImagesPullingTime="3.451454773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:01.26464134 +0000 UTC m=+61.289974875" watchObservedRunningTime="2026-07-23 21:53:02.000897587 +0000 UTC m=+62.026231122" Jul 23 21:53:02.001963 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.001937 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:53:02.152041 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.152001 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerStarted","Data":"064b1e222a286cf66a21857ace79d6b97f63d34daf28e72e947fdad195b2307b"} Jul 23 21:53:02.156220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.156119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"41e93f1ad3a6fc9c746c750e476622764bd0fc51cf4bb4224c2eed8deb1988a5"} Jul 23 21:53:02.156220 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.156203 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"a1b7cdd35afd2aeb982d15b200e21f436be2e652faae60cadd96e2b31611807b"} Jul 23 21:53:02.156420 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.156245 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" event={"ID":"a38c4995-e680-4da3-b8b3-7f528b49d86a","Type":"ContainerStarted","Data":"23c152d7dd6c2220e044dc163ec250bb70f05e1900f8295bd82eebb9f8f1fc09"} Jul 23 21:53:02.156903 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.156881 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:53:02.192181 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.192126 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.388516062 podStartE2EDuration="10.192107117s" totalImagesPullingTime="5.780798044s" totalInitContainerRuntime="22.793011ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:02.190410316 +0000 UTC m=+62.215743853" watchObservedRunningTime="2026-07-23 21:53:02.192107117 +0000 UTC m=+62.217440654" Jul 23 21:53:02.224996 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:02.224898 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" podStartSLOduration=2.752521842 podStartE2EDuration="9.224879189s" totalImagesPullingTime="6.472357347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:02.222392261 +0000 UTC m=+62.247725822" watchObservedRunningTime="2026-07-23 21:53:02.224879189 +0000 UTC m=+62.250212724" Jul 23 21:53:05.175516 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175465 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} Jul 23 21:53:05.175888 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} Jul 23 21:53:05.175888 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} Jul 23 21:53:05.175888 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175602 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} Jul 23 21:53:05.175888 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175626 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} Jul 23 21:53:05.175888 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.175648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerStarted","Data":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} Jul 23 21:53:05.211182 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:05.211101 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.833069507 podStartE2EDuration="8.211071644s" totalImagesPullingTime="3.352054013s" totalInitContainerRuntime="25.948124ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:05.209035381 +0000 UTC m=+65.234368930" watchObservedRunningTime="2026-07-23 21:53:05.211071644 +0000 UTC m=+65.236405181" Jul 23 21:53:06.542314 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:06.542278 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:53:07.566052 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.566013 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:53:07.566509 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.566119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:53:07.566509 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.566176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:53:07.568808 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.568782 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 21:53:07.569702 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.569689 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 21:53:07.569784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.569699 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 21:53:07.578676 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.578654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/235d6a13-4446-4a06-93e4-ae7e9a3634c9-metrics-certs\") pod \"network-metrics-daemon-cznsg\" (UID: \"235d6a13-4446-4a06-93e4-ae7e9a3634c9\") " pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:53:07.578787 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.578750 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da-original-pull-secret\") pod \"global-pull-secret-syncer-b5sb5\" (UID: \"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da\") " pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:53:07.580032 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.580014 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 21:53:07.589133 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.589109 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbnr\" (UniqueName: \"kubernetes.io/projected/a4d14601-2e8f-4cb4-bf07-6b9f72091874-kube-api-access-nvbnr\") pod \"network-check-target-bh2r8\" (UID: \"a4d14601-2e8f-4cb4-bf07-6b9f72091874\") " pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:53:07.692569 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.692531 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5sb5" Jul 23 21:53:07.701120 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.701096 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd2p5\"" Jul 23 21:53:07.709442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.709415 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cznsg" Jul 23 21:53:07.829397 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.829369 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b5sb5"] Jul 23 21:53:07.832165 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:53:07.832140 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab2eb30_dc87_4854_bd4a_dbbc7a68e1da.slice/crio-cedcf099d366ca53f75484d8fa3c35a2096a068e0d3ec78059063f388ef4d626 WatchSource:0}: Error finding container cedcf099d366ca53f75484d8fa3c35a2096a068e0d3ec78059063f388ef4d626: Status 404 returned error can't find the container with id cedcf099d366ca53f75484d8fa3c35a2096a068e0d3ec78059063f388ef4d626 Jul 23 21:53:07.850438 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.850413 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cznsg"] Jul 23 21:53:07.852702 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:53:07.852678 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d6a13_4446_4a06_93e4_ae7e9a3634c9.slice/crio-9e9570dccb923b596de60491eee3e52a6f9469cbfb2870932dd83599370bb16c WatchSource:0}: Error finding container 9e9570dccb923b596de60491eee3e52a6f9469cbfb2870932dd83599370bb16c: Status 404 returned error can't find the container with id 9e9570dccb923b596de60491eee3e52a6f9469cbfb2870932dd83599370bb16c Jul 23 21:53:07.886914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.886895 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2x2\"" Jul 23 21:53:07.895023 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:07.895007 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:53:08.015526 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.015502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh2r8"] Jul 23 21:53:08.017697 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:53:08.017673 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d14601_2e8f_4cb4_bf07_6b9f72091874.slice/crio-82cbb168699a79a60ff8f932420d821ed6bdcf173e2c6416d2307da062423a48 WatchSource:0}: Error finding container 82cbb168699a79a60ff8f932420d821ed6bdcf173e2c6416d2307da062423a48: Status 404 returned error can't find the container with id 82cbb168699a79a60ff8f932420d821ed6bdcf173e2c6416d2307da062423a48 Jul 23 21:53:08.149600 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.149566 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:08.168745 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.168723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6887688fc8-7rt52" Jul 23 21:53:08.190166 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.189080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b5sb5" event={"ID":"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da","Type":"ContainerStarted","Data":"cedcf099d366ca53f75484d8fa3c35a2096a068e0d3ec78059063f388ef4d626"} Jul 23 21:53:08.191788 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.191745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh2r8" event={"ID":"a4d14601-2e8f-4cb4-bf07-6b9f72091874","Type":"ContainerStarted","Data":"82cbb168699a79a60ff8f932420d821ed6bdcf173e2c6416d2307da062423a48"} Jul 23 21:53:08.193138 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:08.193117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cznsg" event={"ID":"235d6a13-4446-4a06-93e4-ae7e9a3634c9","Type":"ContainerStarted","Data":"9e9570dccb923b596de60491eee3e52a6f9469cbfb2870932dd83599370bb16c"} Jul 23 21:53:09.201155 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:09.201008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cznsg" event={"ID":"235d6a13-4446-4a06-93e4-ae7e9a3634c9","Type":"ContainerStarted","Data":"fe904388649b72d6ef196304b65c656bf4c9ca54c993726bf7d4a79657ce2e96"} Jul 23 21:53:09.201155 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:09.201117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cznsg" event={"ID":"235d6a13-4446-4a06-93e4-ae7e9a3634c9","Type":"ContainerStarted","Data":"36805c31d91d9ad4ceb4b3f1fcc528fb3312691e89e6dace828106de8e54422c"} Jul 23 21:53:09.220437 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:09.220337 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cznsg" podStartSLOduration=67.116027238 podStartE2EDuration="1m8.220307378s" totalImagesPullingTime="1.10428014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:09.217755218 +0000 UTC m=+69.243088753" watchObservedRunningTime="2026-07-23 21:53:09.220307378 +0000 UTC m=+69.245640914" Jul 23 21:53:13.045904 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:13.045870 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:53:13.222748 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:13.222715 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh2r8" event={"ID":"a4d14601-2e8f-4cb4-bf07-6b9f72091874","Type":"ContainerStarted","Data":"941062b9e8b336d437f4496c5d6a1545fe422da90914b338280cb9bedc61c596"} Jul 23 21:53:13.222864 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:13.222845 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:53:13.251798 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:13.251738 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bh2r8" podStartSLOduration=67.172904245 podStartE2EDuration="1m12.251718277s" totalImagesPullingTime="5.078814032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:13.248608526 +0000 UTC m=+73.273942060" watchObservedRunningTime="2026-07-23 21:53:13.251718277 +0000 UTC m=+73.277051813" Jul 23 21:53:14.228564 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:14.228526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b5sb5" event={"ID":"1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da","Type":"ContainerStarted","Data":"1230151112a8d849ad84c0acc767e6931ffc1096ec5f25977e5ddb79ea9f3ff4"} Jul 23 21:53:16.310338 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:16.310301 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:53:16.310738 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:16.310379 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:53:18.074178 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.074116 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84c9585844-nsqxb" podUID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" containerName="console" containerID="cri-o://bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716" gracePeriod=15 Jul 23 21:53:18.841291 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.841265 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:53:18.862881 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.862827 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b5sb5" podStartSLOduration=71.587873988 podStartE2EDuration="1m16.862806452s" totalImagesPullingTime="5.274932464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:14.245821942 +0000 UTC m=+74.271155478" watchObservedRunningTime="2026-07-23 21:53:18.862806452 +0000 UTC m=+78.888139987" Jul 23 21:53:18.979215 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979096 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979215 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979178 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979215 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979215 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979475 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979259 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnp2t\" (UniqueName: \"kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979475 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979298 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979475 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979349 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert\") pod \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\" (UID: \"f656b369-2dea-4b32-96f5-2abbfd6d74c5\") " Jul 23 21:53:18.979668 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979580 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config" (OuterVolumeSpecName: "console-config") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:18.979727 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979697 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:18.979803 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.979779 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:18.981559 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.981526 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:18.981559 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.981536 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t" (OuterVolumeSpecName: "kube-api-access-jnp2t") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "kube-api-access-jnp2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:18.981701 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:18.981598 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f656b369-2dea-4b32-96f5-2abbfd6d74c5" (UID: "f656b369-2dea-4b32-96f5-2abbfd6d74c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:19.080714 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080669 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-oauth-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.080714 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080710 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-oauth-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.080714 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080720 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.080714 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080729 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f656b369-2dea-4b32-96f5-2abbfd6d74c5-console-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.081167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080738 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f656b369-2dea-4b32-96f5-2abbfd6d74c5-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.081167 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.080746 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnp2t\" (UniqueName: \"kubernetes.io/projected/f656b369-2dea-4b32-96f5-2abbfd6d74c5-kube-api-access-jnp2t\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:19.254442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.254362 2569 generic.go:356] "Generic (PLEG): container finished" podID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" containerID="bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716" exitCode=0 Jul 23 21:53:19.254442 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.254421 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9585844-nsqxb" Jul 23 21:53:19.254625 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.254448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9585844-nsqxb" event={"ID":"f656b369-2dea-4b32-96f5-2abbfd6d74c5","Type":"ContainerDied","Data":"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716"} Jul 23 21:53:19.254625 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.254547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9585844-nsqxb" event={"ID":"f656b369-2dea-4b32-96f5-2abbfd6d74c5","Type":"ContainerDied","Data":"4d0bac4f330e80fb3cd513c3064cb723272620fc07d506d68930fd5ba5da7cfe"} Jul 23 21:53:19.254625 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.254573 2569 scope.go:117] "RemoveContainer" containerID="bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716" Jul 23 21:53:19.264336 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.264318 2569 scope.go:117] "RemoveContainer" containerID="bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716" Jul 23 21:53:19.264612 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:19.264595 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716\": container with ID starting with bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716 not found: ID does not exist" containerID="bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716" Jul 23 21:53:19.264670 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.264620 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716"} err="failed to get container status \"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716\": rpc error: code = NotFound desc = could not find container \"bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716\": container with ID starting with bb05ef080a4d51608b65f056dcdf15dd1715e3b7593eddd8e089c6966b0b4716 not found: ID does not exist" Jul 23 21:53:19.275506 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.275460 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:53:19.278751 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.278724 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c9585844-nsqxb"] Jul 23 21:53:19.776199 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:19.776171 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" path="/var/lib/kubelet/pods/f656b369-2dea-4b32-96f5-2abbfd6d74c5/volumes" Jul 23 21:53:25.206744 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:25.206602 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:25.284449 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:25.284423 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:25.299302 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:25.299280 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:27.027392 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.027330 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d58c55694-x2sp2" podUID="28ba28d1-fcce-46af-809e-4be124bf576c" containerName="console" containerID="cri-o://687df32de931ae60a9784c6f8fec1bc396fa246c43751ba9a71411a61d8d8837" gracePeriod=15 Jul 23 21:53:27.288657 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.288575 2569 generic.go:356] "Generic (PLEG): container finished" podID="28ba28d1-fcce-46af-809e-4be124bf576c" containerID="687df32de931ae60a9784c6f8fec1bc396fa246c43751ba9a71411a61d8d8837" exitCode=0 Jul 23 21:53:27.288812 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.288651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d58c55694-x2sp2" event={"ID":"28ba28d1-fcce-46af-809e-4be124bf576c","Type":"ContainerDied","Data":"687df32de931ae60a9784c6f8fec1bc396fa246c43751ba9a71411a61d8d8837"} Jul 23 21:53:27.819091 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.819064 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:53:27.865247 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865215 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865266 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865285 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865306 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865332 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5gjg\" (UniqueName: \"kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865376 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865677 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865440 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert\") pod \"28ba28d1-fcce-46af-809e-4be124bf576c\" (UID: \"28ba28d1-fcce-46af-809e-4be124bf576c\") " Jul 23 21:53:27.865677 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865654 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config" (OuterVolumeSpecName: "console-config") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:27.865780 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865679 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:27.865780 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865757 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca" (OuterVolumeSpecName: "service-ca") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:27.865868 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865809 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-console-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.865868 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.865828 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-oauth-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.866164 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.866141 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:27.867832 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.867805 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:27.867928 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.867822 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg" (OuterVolumeSpecName: "kube-api-access-h5gjg") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "kube-api-access-h5gjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:27.867928 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.867881 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "28ba28d1-fcce-46af-809e-4be124bf576c" (UID: "28ba28d1-fcce-46af-809e-4be124bf576c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:27.966906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.966841 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-oauth-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.966906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.966865 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.966906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.966874 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gjg\" (UniqueName: \"kubernetes.io/projected/28ba28d1-fcce-46af-809e-4be124bf576c-kube-api-access-h5gjg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.966906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.966883 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ba28d1-fcce-46af-809e-4be124bf576c-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:27.966906 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:27.966892 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ba28d1-fcce-46af-809e-4be124bf576c-console-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:28.293986 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:28.293911 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d58c55694-x2sp2" Jul 23 21:53:28.293986 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:28.293944 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d58c55694-x2sp2" event={"ID":"28ba28d1-fcce-46af-809e-4be124bf576c","Type":"ContainerDied","Data":"fc0f1380429fec0c098328fd613aadf40c443261614d7676f00d544770488bc4"} Jul 23 21:53:28.294401 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:28.294003 2569 scope.go:117] "RemoveContainer" containerID="687df32de931ae60a9784c6f8fec1bc396fa246c43751ba9a71411a61d8d8837" Jul 23 21:53:28.315297 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:28.315269 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:53:28.321350 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:28.321326 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d58c55694-x2sp2"] Jul 23 21:53:29.775827 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:29.775794 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ba28d1-fcce-46af-809e-4be124bf576c" path="/var/lib/kubelet/pods/28ba28d1-fcce-46af-809e-4be124bf576c/volumes" Jul 23 21:53:36.315727 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:36.315697 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:53:36.319539 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:36.319514 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5cc8ffdcb8-k42xl" Jul 23 21:53:38.066454 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.066389 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5df5474b-r8qfh" podUID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" containerName="console" containerID="cri-o://12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a" gracePeriod=15 Jul 23 21:53:38.332135 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.332109 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:53:38.339376 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.339352 2569 generic.go:356] "Generic (PLEG): container finished" podID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" containerID="12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a" exitCode=0 Jul 23 21:53:38.339476 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.339396 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5df5474b-r8qfh" Jul 23 21:53:38.339476 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.339421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5df5474b-r8qfh" event={"ID":"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9","Type":"ContainerDied","Data":"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a"} Jul 23 21:53:38.339476 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.339457 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5df5474b-r8qfh" event={"ID":"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9","Type":"ContainerDied","Data":"1a43ef2ee2a36419fada79a93a18f5b77a8cdff835fa3fd6afc57cc205182576"} Jul 23 21:53:38.339476 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.339474 2569 scope.go:117] "RemoveContainer" containerID="12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a" Jul 23 21:53:38.347208 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.347189 2569 scope.go:117] "RemoveContainer" containerID="12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a" Jul 23 21:53:38.347454 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:38.347435 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a\": container with ID starting with 12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a not found: ID does not exist" containerID="12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a" Jul 23 21:53:38.347525 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.347461 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a"} err="failed to get container status \"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a\": rpc error: code = NotFound desc = could not find container \"12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a\": container with ID starting with 12488dbc6d64113a672dc5e7d3dd2a648b91a10f7df26ece8e07c4d567a7226a not found: ID does not exist" Jul 23 21:53:38.459509 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459437 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6q2z\" (UniqueName: \"kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459509 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459515 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459544 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459562 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459629 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.459755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459645 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert\") pod \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\" (UID: \"cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9\") " Jul 23 21:53:38.460003 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.459909 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config" (OuterVolumeSpecName: "console-config") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:38.460121 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.460092 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca" (OuterVolumeSpecName: "service-ca") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:38.460191 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.460128 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:38.460191 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.460139 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:38.461862 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.461835 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:38.461951 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.461833 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z" (OuterVolumeSpecName: "kube-api-access-m6q2z") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "kube-api-access-m6q2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:38.461951 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.461903 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" (UID: "cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:38.560583 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560549 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q2z\" (UniqueName: \"kubernetes.io/projected/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-kube-api-access-m6q2z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560583 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560578 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560583 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560587 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560595 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-oauth-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560604 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560612 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-oauth-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.560784 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.560621 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9-console-serving-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:38.660733 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.660701 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:53:38.664803 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:38.664782 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5df5474b-r8qfh"] Jul 23 21:53:39.775342 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:39.775299 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" path="/var/lib/kubelet/pods/cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9/volumes" Jul 23 21:53:41.227864 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.227829 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228116 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="alertmanager" containerID="cri-o://adfb9bc8fb2ef9ec690034e714cbb58791e711ae3b5e3444b0134cc2c1238429" gracePeriod=120 Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228177 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-metric" containerID="cri-o://6fb40a030cf80c505d950f0b34295be2359317f9e137ba434df274a434f5e091" gracePeriod=120 Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228201 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="config-reloader" containerID="cri-o://950b59fdb7135c688308c7aca4b17299cae92e2885148ea94aeef0b7e0091696" gracePeriod=120 Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228213 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-web" containerID="cri-o://93244bd0df218bb501807f318ceb8edf52629095931daac18730a30bac094779" gracePeriod=120 Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228233 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy" containerID="cri-o://3cd7a3377fd2fa95a694e9e70b2676261dcad437002dbf9af5778422c25619c3" gracePeriod=120 Jul 23 21:53:41.228373 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.228270 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="prom-label-proxy" containerID="cri-o://064b1e222a286cf66a21857ace79d6b97f63d34daf28e72e947fdad195b2307b" gracePeriod=120 Jul 23 21:53:41.357423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357392 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="064b1e222a286cf66a21857ace79d6b97f63d34daf28e72e947fdad195b2307b" exitCode=0 Jul 23 21:53:41.357423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357418 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="6fb40a030cf80c505d950f0b34295be2359317f9e137ba434df274a434f5e091" exitCode=0 Jul 23 21:53:41.357423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357424 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="3cd7a3377fd2fa95a694e9e70b2676261dcad437002dbf9af5778422c25619c3" exitCode=0 Jul 23 21:53:41.357423 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357429 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="93244bd0df218bb501807f318ceb8edf52629095931daac18730a30bac094779" exitCode=0 Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357436 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="950b59fdb7135c688308c7aca4b17299cae92e2885148ea94aeef0b7e0091696" exitCode=0 Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357443 2569 generic.go:356] "Generic (PLEG): container finished" podID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerID="adfb9bc8fb2ef9ec690034e714cbb58791e711ae3b5e3444b0134cc2c1238429" exitCode=0 Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"064b1e222a286cf66a21857ace79d6b97f63d34daf28e72e947fdad195b2307b"} Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357548 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"6fb40a030cf80c505d950f0b34295be2359317f9e137ba434df274a434f5e091"} Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"3cd7a3377fd2fa95a694e9e70b2676261dcad437002dbf9af5778422c25619c3"} Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357601 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"93244bd0df218bb501807f318ceb8edf52629095931daac18730a30bac094779"} Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"950b59fdb7135c688308c7aca4b17299cae92e2885148ea94aeef0b7e0091696"} Jul 23 21:53:41.357660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.357647 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"adfb9bc8fb2ef9ec690034e714cbb58791e711ae3b5e3444b0134cc2c1238429"} Jul 23 21:53:41.496728 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.496702 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:41.584894 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.584867 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhr78\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.584894 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.584900 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585114 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.584931 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585114 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.584955 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585114 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585041 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585119 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585139 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585156 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585178 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585209 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585266 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585246 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585584 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585584 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume\") pod \"af3d37d7-331f-4c97-8f52-037c32e983b9\" (UID: \"af3d37d7-331f-4c97-8f52-037c32e983b9\") " Jul 23 21:53:41.585584 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585325 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:41.585584 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585405 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:41.585775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585615 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-metrics-client-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.585775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.585635 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.588066 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588034 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.588066 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588045 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78" (OuterVolumeSpecName: "kube-api-access-jhr78") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "kube-api-access-jhr78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:41.588259 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588109 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:41.588474 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588427 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out" (OuterVolumeSpecName: "config-out") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 21:53:41.588584 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588466 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 21:53:41.588771 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.588735 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.590075 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.590051 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.590313 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.590290 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume" (OuterVolumeSpecName: "config-volume") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.590397 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.590366 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.592603 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.592581 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.598508 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.598476 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config" (OuterVolumeSpecName: "web-config") pod "af3d37d7-331f-4c97-8f52-037c32e983b9" (UID: "af3d37d7-331f-4c97-8f52-037c32e983b9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:41.686561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686530 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-tls-assets\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686556 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-main-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686561 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686566 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-web-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686575 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-cluster-tls-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686586 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686596 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-alertmanager-main-db\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686605 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686616 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af3d37d7-331f-4c97-8f52-037c32e983b9-config-out\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686624 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686632 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af3d37d7-331f-4c97-8f52-037c32e983b9-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:41.686766 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:41.686640 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhr78\" (UniqueName: \"kubernetes.io/projected/af3d37d7-331f-4c97-8f52-037c32e983b9-kube-api-access-jhr78\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:42.364801 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.364765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af3d37d7-331f-4c97-8f52-037c32e983b9","Type":"ContainerDied","Data":"028493eacf6cfbd1d6757509246784e03953bfad2f6d8023175672ebfa63ea93"} Jul 23 21:53:42.365244 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.364839 2569 scope.go:117] "RemoveContainer" containerID="064b1e222a286cf66a21857ace79d6b97f63d34daf28e72e947fdad195b2307b" Jul 23 21:53:42.365244 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.364840 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.372291 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.372273 2569 scope.go:117] "RemoveContainer" containerID="6fb40a030cf80c505d950f0b34295be2359317f9e137ba434df274a434f5e091" Jul 23 21:53:42.378890 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.378876 2569 scope.go:117] "RemoveContainer" containerID="3cd7a3377fd2fa95a694e9e70b2676261dcad437002dbf9af5778422c25619c3" Jul 23 21:53:42.385453 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.385433 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:42.386549 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.386533 2569 scope.go:117] "RemoveContainer" containerID="93244bd0df218bb501807f318ceb8edf52629095931daac18730a30bac094779" Jul 23 21:53:42.390241 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.390221 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:42.393569 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.393546 2569 scope.go:117] "RemoveContainer" containerID="950b59fdb7135c688308c7aca4b17299cae92e2885148ea94aeef0b7e0091696" Jul 23 21:53:42.400076 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.400061 2569 scope.go:117] "RemoveContainer" containerID="adfb9bc8fb2ef9ec690034e714cbb58791e711ae3b5e3444b0134cc2c1238429" Jul 23 21:53:42.406617 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.406600 2569 scope.go:117] "RemoveContainer" containerID="505c6504ea6f80a2acd67c7b626525e7ba59db5ced9c9fddbf25720cea10c74d" Jul 23 21:53:42.419273 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419253 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:42.419610 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419594 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-web" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419612 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-web" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419626 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="config-reloader" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419635 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="config-reloader" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419656 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ba28d1-fcce-46af-809e-4be124bf576c" containerName="console" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419664 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ba28d1-fcce-46af-809e-4be124bf576c" containerName="console" Jul 23 21:53:42.419678 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419675 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="init-config-reloader" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419684 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="init-config-reloader" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419699 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="alertmanager" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419708 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="alertmanager" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419721 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419729 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419737 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-metric" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419746 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-metric" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419758 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419768 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419778 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419786 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419799 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="prom-label-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419808 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="prom-label-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419883 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-metric" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419898 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="alertmanager" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419907 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8ffb77-d3bb-48f0-93c8-a2b7cae67da9" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419917 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ba28d1-fcce-46af-809e-4be124bf576c" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419925 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419934 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="kube-rbac-proxy-web" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419943 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="prom-label-proxy" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419953 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f656b369-2dea-4b32-96f5-2abbfd6d74c5" containerName="console" Jul 23 21:53:42.419961 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.419961 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" containerName="config-reloader" Jul 23 21:53:42.425637 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.425619 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.428222 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428198 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 21:53:42.428634 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m9frt\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428631 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428659 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428631 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 21:53:42.428767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428733 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 21:53:42.429227 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.428923 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 21:53:42.434072 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.433904 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 21:53:42.436327 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.436307 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:42.494424 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494424 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-web-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494424 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrjg\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-kube-api-access-pcrjg\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494628 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-config-out\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.494869 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.494687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595278 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-web-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595451 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrjg\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-kube-api-access-pcrjg\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-config-out\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.595826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.596077 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.595854 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.596719 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.596692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.597078 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.597050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a624481-7a31-4e43-8e65-434b23781bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.598539 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.598454 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.598645 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.598564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a624481-7a31-4e43-8e65-434b23781bef-config-out\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.598704 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.598674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.599038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.599013 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.599224 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.599205 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.599298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.599233 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.599394 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.599378 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-web-config\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.599651 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.599626 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.600094 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.600076 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a624481-7a31-4e43-8e65-434b23781bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.604145 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.604121 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrjg\" (UniqueName: \"kubernetes.io/projected/3a624481-7a31-4e43-8e65-434b23781bef-kube-api-access-pcrjg\") pod \"alertmanager-main-0\" (UID: \"3a624481-7a31-4e43-8e65-434b23781bef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.736250 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.736214 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 21:53:42.901638 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:42.901614 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 21:53:42.903524 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:53:42.903499 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a624481_7a31_4e43_8e65_434b23781bef.slice/crio-fd67f6d283c0ebf95d90202ee57d4eb84b189d565b8a4489f37d7e9b41be840d WatchSource:0}: Error finding container fd67f6d283c0ebf95d90202ee57d4eb84b189d565b8a4489f37d7e9b41be840d: Status 404 returned error can't find the container with id fd67f6d283c0ebf95d90202ee57d4eb84b189d565b8a4489f37d7e9b41be840d Jul 23 21:53:43.370527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:43.370418 2569 generic.go:356] "Generic (PLEG): container finished" podID="3a624481-7a31-4e43-8e65-434b23781bef" containerID="26fe31cf67c63531ab9d41fba5e85309cf78b8fee64818cd616d701914ed225d" exitCode=0 Jul 23 21:53:43.370527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:43.370504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerDied","Data":"26fe31cf67c63531ab9d41fba5e85309cf78b8fee64818cd616d701914ed225d"} Jul 23 21:53:43.370996 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:43.370597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"fd67f6d283c0ebf95d90202ee57d4eb84b189d565b8a4489f37d7e9b41be840d"} Jul 23 21:53:43.776801 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:43.776769 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af3d37d7-331f-4c97-8f52-037c32e983b9" path="/var/lib/kubelet/pods/af3d37d7-331f-4c97-8f52-037c32e983b9/volumes" Jul 23 21:53:44.231887 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.231808 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bh2r8" Jul 23 21:53:44.379377 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"504e6d940bd5fc8ec86a57528f8b49de9316539a29070d7e379302abdefb231c"} Jul 23 21:53:44.379759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"d6ee2e9690473a7c1ba1c99f34f04be430671a93a232a2dd192d71d49468e0e8"} Jul 23 21:53:44.379759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"7496dd5d7b8cfbc4a418860722e8cecf4e9ef1eb99404aec9664044adbda11fd"} Jul 23 21:53:44.379759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"dd663837b70f6b2255f4fdb306c9f528d83e753b04a8b952527644cd6e4c785b"} Jul 23 21:53:44.379759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379516 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"90061e52fdbdd74b8a2c75e6cdba1aa22ce976cf383e38e54700acdd51b66812"} Jul 23 21:53:44.379759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.379543 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a624481-7a31-4e43-8e65-434b23781bef","Type":"ContainerStarted","Data":"5f19b372cc1f5663e0bc53d948cbdc8ead36816f48e329c19b585a9696322227"} Jul 23 21:53:44.410316 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:44.410232 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.390434109 podStartE2EDuration="2.41021293s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.778821ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:44.409847923 +0000 UTC m=+104.435181457" watchObservedRunningTime="2026-07-23 21:53:44.41021293 +0000 UTC m=+104.435546466" Jul 23 21:53:45.482694 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.482655 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:45.483242 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.483169 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="prometheus" containerID="cri-o://41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" gracePeriod=600 Jul 23 21:53:45.483579 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.483536 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy" containerID="cri-o://e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" gracePeriod=600 Jul 23 21:53:45.484874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.483689 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-web" containerID="cri-o://1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" gracePeriod=600 Jul 23 21:53:45.484874 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.483718 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="thanos-sidecar" containerID="cri-o://6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" gracePeriod=600 Jul 23 21:53:45.485116 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.483735 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="config-reloader" containerID="cri-o://ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" gracePeriod=600 Jul 23 21:53:45.485116 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.484079 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" gracePeriod=600 Jul 23 21:53:45.764802 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.764775 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:45.827355 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827367 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827407 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827510 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827457 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827519 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827545 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827568 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827593 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827624 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827671 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827683 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827710 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827736 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827769 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827796 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827889 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827918 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.827956 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827958 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.828420 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827973 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.828420 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.827984 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0\") pod \"66d736a7-e962-46d0-bef4-8e7365b2381f\" (UID: \"66d736a7-e962-46d0-bef4-8e7365b2381f\") " Jul 23 21:53:45.828420 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.828289 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-metrics-client-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.829374 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.829344 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.830207 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.830175 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.830541 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.830514 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:45.830626 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.830604 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.831261 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.830968 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.831261 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.831142 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.831746 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.831628 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.832015 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.831913 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 21:53:45.832286 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.832261 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.832431 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.832407 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.832848 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.832651 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config" (OuterVolumeSpecName: "config") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.832939 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.832894 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.833078 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.833054 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 21:53:45.833767 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.833372 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.833900 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.833842 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr" (OuterVolumeSpecName: "kube-api-access-fgtqr") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "kube-api-access-fgtqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 21:53:45.835650 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.835586 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out" (OuterVolumeSpecName: "config-out") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 21:53:45.835650 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.835623 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.835778 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.835711 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.844233 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.844211 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config" (OuterVolumeSpecName: "web-config") pod "66d736a7-e962-46d0-bef4-8e7365b2381f" (UID: "66d736a7-e962-46d0-bef4-8e7365b2381f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 21:53:45.929337 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929302 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-web-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929337 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929332 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929347 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929360 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929372 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-metrics-client-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929383 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929395 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929407 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-tls-assets\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929418 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-grpc-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929429 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-kube-rbac-proxy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929440 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-config\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929451 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929465 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929500 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66d736a7-e962-46d0-bef4-8e7365b2381f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929515 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929528 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-config-out\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929540 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-k8s-db\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.929555 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929554 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d736a7-e962-46d0-bef4-8e7365b2381f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:45.930006 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:45.929566 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/66d736a7-e962-46d0-bef4-8e7365b2381f-kube-api-access-fgtqr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 21:53:46.392995 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.392956 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" exitCode=0 Jul 23 21:53:46.392995 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.392988 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" exitCode=0 Jul 23 21:53:46.392995 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.392996 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" exitCode=0 Jul 23 21:53:46.392995 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393004 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" exitCode=0 Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393012 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" exitCode=0 Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393019 2569 generic.go:356] "Generic (PLEG): container finished" podID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" exitCode=0 Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393048 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393198 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393209 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393239 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393279 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} Jul 23 21:53:46.393351 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.393319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66d736a7-e962-46d0-bef4-8e7365b2381f","Type":"ContainerDied","Data":"4eb53867e15102a37c40c02219aa8afaddbcdb8cb04423f27ed9788a1b0ae632"} Jul 23 21:53:46.401838 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.401820 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.408778 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.408761 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.415635 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.415618 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.417774 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.417753 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:46.422951 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.422933 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.429935 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.429913 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:46.430190 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.430169 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.437174 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.437159 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.443544 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.443528 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.443782 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.443765 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.443845 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.443790 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.443845 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.443806 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.444037 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.444019 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.444084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444041 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.444084 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444055 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.444272 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.444251 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.444336 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444280 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.444336 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444304 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.444566 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.444551 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.444633 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444572 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.444633 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444590 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.444848 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.444831 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.444905 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444852 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.444905 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.444866 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.445073 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.445058 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.445109 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445077 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.445109 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445090 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.445288 ip-10-0-142-110 kubenswrapper[2569]: E0723 21:53:46.445272 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.445328 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445292 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.445328 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445305 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.445518 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445497 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.445518 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445518 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.445744 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445727 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.445796 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445745 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.445979 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445963 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.446016 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.445980 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.446217 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446201 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.446257 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446218 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.446430 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446412 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.446471 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446431 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.446652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446635 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.446652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446652 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.446830 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446816 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.446830 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.446829 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.447038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447022 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.447038 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447038 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.447263 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447244 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.447338 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447265 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.447525 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447506 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.447581 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447526 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.447738 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447720 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.447808 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447740 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.447974 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447939 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.448019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.447976 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.448175 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448157 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.448245 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448177 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.448413 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448383 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.448455 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448414 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.448615 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448600 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.448658 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448615 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.448811 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448791 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.448872 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448813 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.449004 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.448989 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.449054 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449003 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.449211 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449194 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.449277 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449212 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.449412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449395 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.449453 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449414 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.449624 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449609 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.449663 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449624 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.449826 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449809 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.449890 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.449828 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.450074 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450058 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.450123 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450074 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.450261 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450245 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.450261 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450260 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.450518 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450493 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.450602 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450519 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.450887 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450771 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.450887 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.450794 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.451083 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451060 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.451123 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451086 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.451358 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451341 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.451402 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451359 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.451614 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451596 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.451662 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451614 2569 scope.go:117] "RemoveContainer" containerID="0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc" Jul 23 21:53:46.451867 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451844 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc"} err="failed to get container status \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": rpc error: code = NotFound desc = could not find container \"0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc\": container with ID starting with 0ec72eeda41cf3bda5c690fa5b181f3954ad86cddee61473b039c136dea3c6cc not found: ID does not exist" Jul 23 21:53:46.451912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.451869 2569 scope.go:117] "RemoveContainer" containerID="e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f" Jul 23 21:53:46.452085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452069 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f"} err="failed to get container status \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": rpc error: code = NotFound desc = could not find container \"e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f\": container with ID starting with e6e020e64a65c7fe1056f99342c30f932423cb3ee1afcde9d9a568ae4c92230f not found: ID does not exist" Jul 23 21:53:46.452085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452085 2569 scope.go:117] "RemoveContainer" containerID="1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9" Jul 23 21:53:46.452314 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452296 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9"} err="failed to get container status \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": rpc error: code = NotFound desc = could not find container \"1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9\": container with ID starting with 1b19f7108f7a00a69dffbaad76bf88e0fda417d084fb7ce9ecc515b006f342f9 not found: ID does not exist" Jul 23 21:53:46.452355 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452314 2569 scope.go:117] "RemoveContainer" containerID="6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d" Jul 23 21:53:46.452550 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452534 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d"} err="failed to get container status \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": rpc error: code = NotFound desc = could not find container \"6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d\": container with ID starting with 6fb927af1ba5e61964674f0aadb2d257bb7f50d57508b7a3e9b39954a1951e2d not found: ID does not exist" Jul 23 21:53:46.452550 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452549 2569 scope.go:117] "RemoveContainer" containerID="ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c" Jul 23 21:53:46.452775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452756 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c"} err="failed to get container status \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": rpc error: code = NotFound desc = could not find container \"ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c\": container with ID starting with ea6d263cc9bd897d17032e7c7cc8bf972c6cce83a3cd1779cf4f3b819a2a407c not found: ID does not exist" Jul 23 21:53:46.452829 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452776 2569 scope.go:117] "RemoveContainer" containerID="41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3" Jul 23 21:53:46.452974 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452959 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3"} err="failed to get container status \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": rpc error: code = NotFound desc = could not find container \"41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3\": container with ID starting with 41e17d055e0fd7792aec6f626f128b9e3c17da26107726941c298526ccd857c3 not found: ID does not exist" Jul 23 21:53:46.453014 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.452975 2569 scope.go:117] "RemoveContainer" containerID="f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998" Jul 23 21:53:46.453158 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.453142 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998"} err="failed to get container status \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": rpc error: code = NotFound desc = could not find container \"f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998\": container with ID starting with f8658951863da4b2235631f577ec12317d62479cce3335abc3ef2eddaf647998 not found: ID does not exist" Jul 23 21:53:46.460307 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460287 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:46.460622 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460610 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy" Jul 23 21:53:46.460660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460627 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy" Jul 23 21:53:46.460660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460642 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="thanos-sidecar" Jul 23 21:53:46.460660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460648 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="thanos-sidecar" Jul 23 21:53:46.460660 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460660 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="init-config-reloader" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460665 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="init-config-reloader" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460672 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-thanos" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460677 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-thanos" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460689 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="prometheus" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460694 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="prometheus" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460704 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="config-reloader" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460712 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="config-reloader" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460719 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-web" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460724 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-web" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460764 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="config-reloader" Jul 23 21:53:46.460775 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460772 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="prometheus" Jul 23 21:53:46.461049 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460784 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-web" Jul 23 21:53:46.461049 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460791 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy-thanos" Jul 23 21:53:46.461049 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460797 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="thanos-sidecar" Jul 23 21:53:46.461049 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.460803 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" containerName="kube-rbac-proxy" Jul 23 21:53:46.491805 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.491781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:46.492144 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.491904 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.495019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.494997 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 21:53:46.495019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495016 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 21:53:46.495223 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495127 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 21:53:46.495223 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495128 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-36ire8h9kjgum\"" Jul 23 21:53:46.495325 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495312 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 21:53:46.495375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495328 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 21:53:46.495375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495340 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 21:53:46.495375 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495351 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 21:53:46.495537 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j9zpm\"" Jul 23 21:53:46.495537 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495342 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 21:53:46.495537 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495316 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 21:53:46.495772 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495743 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 21:53:46.495918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495774 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 21:53:46.495918 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.495794 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 21:53:46.498202 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.498181 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 21:53:46.505904 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.503400 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 21:53:46.635883 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.635845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.635883 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.635883 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.635908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.635962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.635992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636085 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4g8w\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-kube-api-access-c4g8w\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636236 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.636610 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.636408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737165 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737165 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737165 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4g8w\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-kube-api-access-c4g8w\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737412 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737387 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.737992 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.737938 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738378 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.738718 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.738397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.739066 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.739040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.739334 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.739313 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.739970 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.739944 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740145 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740115 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bcca40-0c2c-4c30-b06a-731c71077363-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740263 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740345 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740317 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740418 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740363 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740474 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.740811 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.740786 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.741188 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.741098 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.741574 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.741318 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.741574 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.741336 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bcca40-0c2c-4c30-b06a-731c71077363-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.741574 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.741336 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.742505 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.742456 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.742749 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.742730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-config\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.743127 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.743102 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.743227 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.743133 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.743587 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.743571 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bcca40-0c2c-4c30-b06a-731c71077363-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.746265 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.746244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4g8w\" (UniqueName: \"kubernetes.io/projected/c4bcca40-0c2c-4c30-b06a-731c71077363-kube-api-access-c4g8w\") pod \"prometheus-k8s-0\" (UID: \"c4bcca40-0c2c-4c30-b06a-731c71077363\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.802816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.802785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:53:46.959107 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:46.959076 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 21:53:46.959720 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:53:46.959696 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bcca40_0c2c_4c30_b06a_731c71077363.slice/crio-66b99a4e719822d64f558410eee46981f1122e66471d2b6a2041e97ed7a1c7b6 WatchSource:0}: Error finding container 66b99a4e719822d64f558410eee46981f1122e66471d2b6a2041e97ed7a1c7b6: Status 404 returned error can't find the container with id 66b99a4e719822d64f558410eee46981f1122e66471d2b6a2041e97ed7a1c7b6 Jul 23 21:53:47.398870 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:47.398840 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4bcca40-0c2c-4c30-b06a-731c71077363" containerID="5d288c4206d8fa472679ee6aa48dbac8bfe25ceaedde9c370203dc6ed9ce738c" exitCode=0 Jul 23 21:53:47.399116 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:47.398933 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerDied","Data":"5d288c4206d8fa472679ee6aa48dbac8bfe25ceaedde9c370203dc6ed9ce738c"} Jul 23 21:53:47.399116 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:47.399011 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"66b99a4e719822d64f558410eee46981f1122e66471d2b6a2041e97ed7a1c7b6"} Jul 23 21:53:47.779738 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:47.779705 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d736a7-e962-46d0-bef4-8e7365b2381f" path="/var/lib/kubelet/pods/66d736a7-e962-46d0-bef4-8e7365b2381f/volumes" Jul 23 21:53:48.407146 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"96f10488295d305ddb2beadb46ba35e0cddd8a30dfdfcbed44fe4d8368b2bb0b"} Jul 23 21:53:48.407322 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407205 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"199cbb911de46d8aa9cd9ef845ccb9fed2cfa6e51e9d68ac8a081b39d5c8d1d0"} Jul 23 21:53:48.407322 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"5aa402ed4d9af8916c2824d8c4c8ee71e8da57709db1645d66f6274ec25f7872"} Jul 23 21:53:48.407322 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"f19747ba488f98537298ed4aa60444929ef4df0b5b6c58014aca026136504d84"} Jul 23 21:53:48.407322 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"b984a72dcd5c59dc6152eb8e703a8f4e3c24a363ebe3090f5deea7ce47ce5c32"} Jul 23 21:53:48.407511 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.407326 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bcca40-0c2c-4c30-b06a-731c71077363","Type":"ContainerStarted","Data":"f41c420563754653150a77a10d54f449e1034323795bdcb5edc75b61cdd01de3"} Jul 23 21:53:48.437057 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:48.436910 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.419760839 podStartE2EDuration="2.43689062s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.129781ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:48.435656162 +0000 UTC m=+108.460989724" watchObservedRunningTime="2026-07-23 21:53:48.43689062 +0000 UTC m=+108.462224155" Jul 23 21:53:51.803076 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:53:51.803047 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:54:46.803825 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:54:46.803789 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:54:46.819274 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:54:46.819248 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:54:47.658459 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:54:47.658425 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 21:55:17.125105 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.125016 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht"] Jul 23 21:55:17.128617 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.128595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.131362 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.131342 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-klvsk\"" Jul 23 21:55:17.131469 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.131381 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 21:55:17.131469 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.131442 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 21:55:17.138092 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.138067 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht"] Jul 23 21:55:17.228624 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.228592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.228799 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.228638 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnzw\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-kube-api-access-swnzw\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.329849 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.329803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnzw\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-kube-api-access-swnzw\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.330024 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.329897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.338427 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.338398 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.338629 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.338608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnzw\" (UniqueName: \"kubernetes.io/projected/60d1f334-c9ed-4fe4-9cfa-65db7da404ab-kube-api-access-swnzw\") pod \"cert-manager-cainjector-559cdd8c87-cl6ht\" (UID: \"60d1f334-c9ed-4fe4-9cfa-65db7da404ab\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.449844 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.449813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" Jul 23 21:55:17.574222 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.574194 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht"] Jul 23 21:55:17.576255 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:55:17.576218 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60d1f334_c9ed_4fe4_9cfa_65db7da404ab.slice/crio-094d2b8a5d9911e38816b81f3be51b839e54b61843986f50f99e7ddefe8199df WatchSource:0}: Error finding container 094d2b8a5d9911e38816b81f3be51b839e54b61843986f50f99e7ddefe8199df: Status 404 returned error can't find the container with id 094d2b8a5d9911e38816b81f3be51b839e54b61843986f50f99e7ddefe8199df Jul 23 21:55:17.755064 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:17.754983 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" event={"ID":"60d1f334-c9ed-4fe4-9cfa-65db7da404ab","Type":"ContainerStarted","Data":"094d2b8a5d9911e38816b81f3be51b839e54b61843986f50f99e7ddefe8199df"} Jul 23 21:55:21.778764 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:21.778729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" event={"ID":"60d1f334-c9ed-4fe4-9cfa-65db7da404ab","Type":"ContainerStarted","Data":"e4f597ebb1ccb1143eaf298e52272877618341f47a425d3821c443c8a0664d32"} Jul 23 21:55:21.797370 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:21.797322 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cl6ht" podStartSLOduration=1.5769031519999999 podStartE2EDuration="4.797307614s" totalImagesPullingTime="3.220404462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:55:21.796178036 +0000 UTC m=+201.821511573" watchObservedRunningTime="2026-07-23 21:55:21.797307614 +0000 UTC m=+201.822641149" Jul 23 21:55:28.003589 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.003551 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h"] Jul 23 21:55:28.006917 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.006901 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.009604 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.009578 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 21:55:28.010665 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.010647 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 21:55:28.010773 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.010670 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hvztg\"" Jul 23 21:55:28.015105 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.015081 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h"] Jul 23 21:55:28.123709 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.123670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmczq\" (UniqueName: \"kubernetes.io/projected/97d2b7df-c98b-434d-bb35-292e89ec4cc6-kube-api-access-hmczq\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.123878 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.123734 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d2b7df-c98b-434d-bb35-292e89ec4cc6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.224736 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.224698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d2b7df-c98b-434d-bb35-292e89ec4cc6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.224911 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.224782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmczq\" (UniqueName: \"kubernetes.io/projected/97d2b7df-c98b-434d-bb35-292e89ec4cc6-kube-api-access-hmczq\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.225111 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.225091 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d2b7df-c98b-434d-bb35-292e89ec4cc6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.232553 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.232524 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmczq\" (UniqueName: \"kubernetes.io/projected/97d2b7df-c98b-434d-bb35-292e89ec4cc6-kube-api-access-hmczq\") pod \"openshift-lws-operator-fd8ccff4c-dkb4h\" (UID: \"97d2b7df-c98b-434d-bb35-292e89ec4cc6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.316755 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.316665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" Jul 23 21:55:28.440433 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.440407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h"] Jul 23 21:55:28.443252 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:55:28.443221 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d2b7df_c98b_434d_bb35_292e89ec4cc6.slice/crio-9f3eb186ff041fe01b713f87efaa6ec8afb85918f5e78043414b5e8ee06d04f6 WatchSource:0}: Error finding container 9f3eb186ff041fe01b713f87efaa6ec8afb85918f5e78043414b5e8ee06d04f6: Status 404 returned error can't find the container with id 9f3eb186ff041fe01b713f87efaa6ec8afb85918f5e78043414b5e8ee06d04f6 Jul 23 21:55:28.805498 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:28.805452 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" event={"ID":"97d2b7df-c98b-434d-bb35-292e89ec4cc6","Type":"ContainerStarted","Data":"9f3eb186ff041fe01b713f87efaa6ec8afb85918f5e78043414b5e8ee06d04f6"} Jul 23 21:55:31.819768 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:31.819728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" event={"ID":"97d2b7df-c98b-434d-bb35-292e89ec4cc6","Type":"ContainerStarted","Data":"11b9eb6b61bee089024a3cf50bc8ab1e544b3f94548674361e0181eea3d50859"} Jul 23 21:55:31.838019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:31.837968 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dkb4h" podStartSLOduration=2.252414787 podStartE2EDuration="4.837953654s" totalImagesPullingTime="2.585538867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:55:31.836282448 +0000 UTC m=+211.861615985" watchObservedRunningTime="2026-07-23 21:55:31.837953654 +0000 UTC m=+211.863287246" Jul 23 21:55:51.610718 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.610687 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp"] Jul 23 21:55:51.613238 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.613223 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.616637 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.616609 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 21:55:51.616788 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.616741 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 21:55:51.616998 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.616983 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 21:55:51.617325 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.617307 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zt4v2\"" Jul 23 21:55:51.617420 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.617356 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 21:55:51.639720 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.639698 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp"] Jul 23 21:55:51.738366 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.738269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.738664 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.738633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-manifests\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.738781 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.738691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-tmp\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.738781 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.738769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cj89\" (UniqueName: \"kubernetes.io/projected/35547a13-180d-475d-9a6e-934a0090328a-kube-api-access-9cj89\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.738897 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.738868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.839441 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.839652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-manifests\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.839652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-tmp\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.839652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cj89\" (UniqueName: \"kubernetes.io/projected/35547a13-180d-475d-9a6e-934a0090328a-kube-api-access-9cj89\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.839652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.840019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.839991 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-manifests\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.841987 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.841966 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.842118 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.842048 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35547a13-180d-475d-9a6e-934a0090328a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.842215 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.842197 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35547a13-180d-475d-9a6e-934a0090328a-tmp\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.851254 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.851201 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cj89\" (UniqueName: \"kubernetes.io/projected/35547a13-180d-475d-9a6e-934a0090328a-kube-api-access-9cj89\") pod \"opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp\" (UID: \"35547a13-180d-475d-9a6e-934a0090328a\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:51.923761 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:51.923736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:52.059893 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:52.059869 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp"] Jul 23 21:55:52.062289 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:55:52.062250 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35547a13_180d_475d_9a6e_934a0090328a.slice/crio-5c7ff54cde6554872769bb3c039facf9c0891647c1126ee9aae5278134890ad8 WatchSource:0}: Error finding container 5c7ff54cde6554872769bb3c039facf9c0891647c1126ee9aae5278134890ad8: Status 404 returned error can't find the container with id 5c7ff54cde6554872769bb3c039facf9c0891647c1126ee9aae5278134890ad8 Jul 23 21:55:52.912810 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:52.912768 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" event={"ID":"35547a13-180d-475d-9a6e-934a0090328a","Type":"ContainerStarted","Data":"5c7ff54cde6554872769bb3c039facf9c0891647c1126ee9aae5278134890ad8"} Jul 23 21:55:54.923367 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:54.923332 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" event={"ID":"35547a13-180d-475d-9a6e-934a0090328a","Type":"ContainerStarted","Data":"7b5c7db8d9dac392bc0397debecd12299d670b9e9af49d81f7828b5e23d3acc4"} Jul 23 21:55:56.932854 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:56.932818 2569 generic.go:356] "Generic (PLEG): container finished" podID="35547a13-180d-475d-9a6e-934a0090328a" containerID="7b5c7db8d9dac392bc0397debecd12299d670b9e9af49d81f7828b5e23d3acc4" exitCode=0 Jul 23 21:55:56.933280 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:56.932876 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" event={"ID":"35547a13-180d-475d-9a6e-934a0090328a","Type":"ContainerDied","Data":"7b5c7db8d9dac392bc0397debecd12299d670b9e9af49d81f7828b5e23d3acc4"} Jul 23 21:55:57.938971 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:57.938927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" event={"ID":"35547a13-180d-475d-9a6e-934a0090328a","Type":"ContainerStarted","Data":"f0ffaa3b42fb7b2ef8c31a33cb463f917496ad792630ba444a8e62adf133d95d"} Jul 23 21:55:57.939368 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:57.939162 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:55:57.964226 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:55:57.964182 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" podStartSLOduration=2.600277862 podStartE2EDuration="6.964167784s" totalImagesPullingTime="3.289922707s" totalInitContainerRuntime="1.073967215s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:55:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:55:57.961687981 +0000 UTC m=+237.987021518" watchObservedRunningTime="2026-07-23 21:55:57.964167784 +0000 UTC m=+237.989501320" Jul 23 21:56:08.945281 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:08.945250 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp" Jul 23 21:56:18.172727 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.172694 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92"] Jul 23 21:56:18.181950 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.181924 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.185115 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.185090 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jwz6v\"" Jul 23 21:56:18.185295 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.185206 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 21:56:18.185295 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.185233 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 21:56:18.185443 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.185317 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 21:56:18.186279 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.186255 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92"] Jul 23 21:56:18.281415 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.281382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.281601 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.281429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8557a17c-e754-40f7-95af-8aeec255647b-manager-config\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.281601 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.281450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-metrics-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.281601 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.281524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkjh\" (UniqueName: \"kubernetes.io/projected/8557a17c-e754-40f7-95af-8aeec255647b-kube-api-access-plkjh\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.382696 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.382657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8557a17c-e754-40f7-95af-8aeec255647b-manager-config\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.382696 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.382700 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-metrics-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.382952 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.382732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkjh\" (UniqueName: \"kubernetes.io/projected/8557a17c-e754-40f7-95af-8aeec255647b-kube-api-access-plkjh\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.382952 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.382819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.383396 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.383373 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8557a17c-e754-40f7-95af-8aeec255647b-manager-config\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.385457 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.385431 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.385457 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.385443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8557a17c-e754-40f7-95af-8aeec255647b-metrics-cert\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.391324 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.391301 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkjh\" (UniqueName: \"kubernetes.io/projected/8557a17c-e754-40f7-95af-8aeec255647b-kube-api-access-plkjh\") pod \"lws-controller-manager-66b46854f4-vhp92\" (UID: \"8557a17c-e754-40f7-95af-8aeec255647b\") " pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.492517 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.492394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:18.636940 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.636823 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92"] Jul 23 21:56:18.639848 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:56:18.639820 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8557a17c_e754_40f7_95af_8aeec255647b.slice/crio-1ae77b0dc186d1bd4cdc6d1dd589748eba056dc5f20ddbb5fde39c5f507f16c8 WatchSource:0}: Error finding container 1ae77b0dc186d1bd4cdc6d1dd589748eba056dc5f20ddbb5fde39c5f507f16c8: Status 404 returned error can't find the container with id 1ae77b0dc186d1bd4cdc6d1dd589748eba056dc5f20ddbb5fde39c5f507f16c8 Jul 23 21:56:18.724229 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.723261 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d"] Jul 23 21:56:18.728385 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.728357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.730967 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.730934 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 21:56:18.731124 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.731000 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nsg7h\"" Jul 23 21:56:18.731124 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.731013 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 21:56:18.731124 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.730934 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 21:56:18.737847 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.737781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d"] Jul 23 21:56:18.789401 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdft\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-kube-api-access-vhdft\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.789697 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.789813 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.789813 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789756 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.789950 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.790021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.790021 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.789999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.790120 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.790083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.790179 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.790148 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890588 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890588 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.890816 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891099 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890878 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdft\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-kube-api-access-vhdft\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891099 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.890982 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891099 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.891027 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891242 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.891165 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891242 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.891187 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.891592 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.891559 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.893061 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.893035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.893291 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.893275 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.900414 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.900392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdft\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-kube-api-access-vhdft\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:18.900543 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:18.900414 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0e6693d-0015-41f2-bf37-b09cb5aa177e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l4d8d\" (UID: \"b0e6693d-0015-41f2-bf37-b09cb5aa177e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:19.035048 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:19.035008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" event={"ID":"8557a17c-e754-40f7-95af-8aeec255647b","Type":"ContainerStarted","Data":"1ae77b0dc186d1bd4cdc6d1dd589748eba056dc5f20ddbb5fde39c5f507f16c8"} Jul 23 21:56:19.045136 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:19.045080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:19.176606 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:19.176571 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d"] Jul 23 21:56:19.178610 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:56:19.178579 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e6693d_0015_41f2_bf37_b09cb5aa177e.slice/crio-962fdd8c15a59b5c5362a2cadd7015417bf75b2496e544aa680d1f0291f77516 WatchSource:0}: Error finding container 962fdd8c15a59b5c5362a2cadd7015417bf75b2496e544aa680d1f0291f77516: Status 404 returned error can't find the container with id 962fdd8c15a59b5c5362a2cadd7015417bf75b2496e544aa680d1f0291f77516 Jul 23 21:56:20.040876 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.040846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" event={"ID":"b0e6693d-0015-41f2-bf37-b09cb5aa177e","Type":"ContainerStarted","Data":"962fdd8c15a59b5c5362a2cadd7015417bf75b2496e544aa680d1f0291f77516"} Jul 23 21:56:20.853637 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.853604 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82"] Jul 23 21:56:20.857031 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.857009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.861422 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.861399 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 21:56:20.861581 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.861404 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 21:56:20.861581 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.861405 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 21:56:20.861581 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.861410 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-686hb\"" Jul 23 21:56:20.874143 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.874112 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82"] Jul 23 21:56:20.909194 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ede2f35-344b-43a1-94e3-b86f569bb891-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909405 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:20.909655 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:20.909466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtzw\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-kube-api-access-8wtzw\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010515 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ede2f35-344b-43a1-94e3-b86f569bb891-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010697 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010697 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010697 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010697 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtzw\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-kube-api-access-8wtzw\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010917 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.010917 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.010761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.011912 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.011856 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.016738 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.016682 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ede2f35-344b-43a1-94e3-b86f569bb891-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.016877 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.016861 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.017014 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.016975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.021514 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.021468 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.025083 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.025040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtzw\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-kube-api-access-8wtzw\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.026521 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.026460 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ede2f35-344b-43a1-94e3-b86f569bb891-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvs82\" (UID: \"8ede2f35-344b-43a1-94e3-b86f569bb891\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.048988 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.048943 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" event={"ID":"8557a17c-e754-40f7-95af-8aeec255647b","Type":"ContainerStarted","Data":"bdf1b8b2209355366c2595b4c1d3af3d70acb0375e1e520f3871bafa05bea1be"} Jul 23 21:56:21.049127 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.049086 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:56:21.077152 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.077089 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" podStartSLOduration=1.648120836 podStartE2EDuration="3.077050812s" totalImagesPullingTime="1.428929976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:56:21.07662914 +0000 UTC m=+261.101962687" watchObservedRunningTime="2026-07-23 21:56:21.077050812 +0000 UTC m=+261.102384349" Jul 23 21:56:21.166463 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.166421 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:21.532159 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:21.532081 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82"] Jul 23 21:56:21.535455 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:56:21.535420 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ede2f35_344b_43a1_94e3_b86f569bb891.slice/crio-c250e3ca5d2d2e4bc274578ffc3741ca42cd2e3cea15e64b489977caf1f32675 WatchSource:0}: Error finding container c250e3ca5d2d2e4bc274578ffc3741ca42cd2e3cea15e64b489977caf1f32675: Status 404 returned error can't find the container with id c250e3ca5d2d2e4bc274578ffc3741ca42cd2e3cea15e64b489977caf1f32675 Jul 23 21:56:22.065288 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:22.065190 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" event={"ID":"8ede2f35-344b-43a1-94e3-b86f569bb891","Type":"ContainerStarted","Data":"c250e3ca5d2d2e4bc274578ffc3741ca42cd2e3cea15e64b489977caf1f32675"} Jul 23 21:56:23.051745 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:23.051702 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:56:23.051820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:23.051783 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:56:23.051820 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:23.051813 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:56:24.076537 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:24.076471 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" event={"ID":"b0e6693d-0015-41f2-bf37-b09cb5aa177e","Type":"ContainerStarted","Data":"82b0dcb0cc87b5d480a219e143c19b37aa0389a92889b8dc48b1772f275268e4"} Jul 23 21:56:24.103875 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:24.103821 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" podStartSLOduration=2.233025018 podStartE2EDuration="6.103806101s" totalImagesPullingTime="3.870781083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:56:24.102536802 +0000 UTC m=+264.127870344" watchObservedRunningTime="2026-07-23 21:56:24.103806101 +0000 UTC m=+264.129139637" Jul 23 21:56:25.045425 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:25.045386 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:25.049957 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:25.049932 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:25.082429 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:25.082396 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:25.083298 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:25.083279 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l4d8d" Jul 23 21:56:27.139943 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:27.139911 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:56:27.140200 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:27.139974 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:56:28.098253 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:28.098208 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" event={"ID":"8ede2f35-344b-43a1-94e3-b86f569bb891","Type":"ContainerStarted","Data":"647b699363e8d258a2b49249ca8dd7f8b1a820dc03b527b8f748d14e42f25562"} Jul 23 21:56:28.098457 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:28.098435 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:28.100112 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:28.100089 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" Jul 23 21:56:28.120751 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:28.120694 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvs82" podStartSLOduration=2.518976994 podStartE2EDuration="8.120676048s" totalImagesPullingTime="5.601699054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:56:28.118501995 +0000 UTC m=+268.143835523" watchObservedRunningTime="2026-07-23 21:56:28.120676048 +0000 UTC m=+268.146009586" Jul 23 21:56:32.067192 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:56:32.067161 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66b46854f4-vhp92" Jul 23 21:57:01.632170 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:57:01.632142 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:57:01.638207 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:57:01.638169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 21:57:01.639933 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:57:01.639909 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 21:58:47.106287 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.106248 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm"] Jul 23 21:58:47.110019 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.109997 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.113609 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.113591 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xd9rq\"" Jul 23 21:58:47.125937 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.125907 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm"] Jul 23 21:58:47.252379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252379 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f47074-5175-4c4b-9398-8b5f7ab490b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252545 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252667 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252834 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.252834 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.252714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmd7\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-kube-api-access-pzmd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353652 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmd7\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-kube-api-access-pzmd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f47074-5175-4c4b-9398-8b5f7ab490b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.353914 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.353919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.354073 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354204 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.354157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354387 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.354315 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354440 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.354392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.354440 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.354420 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f47074-5175-4c4b-9398-8b5f7ab490b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.356129 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.356100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.356495 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.356433 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.361738 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.361718 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.362355 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.362335 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmd7\" (UniqueName: \"kubernetes.io/projected/90f47074-5175-4c4b-9398-8b5f7ab490b8-kube-api-access-pzmd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-mltxm\" (UID: \"90f47074-5175-4c4b-9398-8b5f7ab490b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.420759 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.420727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:47.558607 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.558580 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm"] Jul 23 21:58:47.560212 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:58:47.560183 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f47074_5175_4c4b_9398_8b5f7ab490b8.slice/crio-a6ef16ba83f6a6cedfc815d36278f737c6b15b752180dfe203f3c52e0cbada68 WatchSource:0}: Error finding container a6ef16ba83f6a6cedfc815d36278f737c6b15b752180dfe203f3c52e0cbada68: Status 404 returned error can't find the container with id a6ef16ba83f6a6cedfc815d36278f737c6b15b752180dfe203f3c52e0cbada68 Jul 23 21:58:47.562392 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.562352 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:58:47.562470 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.562435 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:58:47.562527 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.562498 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 21:58:47.737524 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.737460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" event={"ID":"90f47074-5175-4c4b-9398-8b5f7ab490b8","Type":"ContainerStarted","Data":"96460361048dbc194b003711e1a1ae3f5851828d565c8116ed92e6141d1b203d"} Jul 23 21:58:47.737690 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.737554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" event={"ID":"90f47074-5175-4c4b-9398-8b5f7ab490b8","Type":"ContainerStarted","Data":"a6ef16ba83f6a6cedfc815d36278f737c6b15b752180dfe203f3c52e0cbada68"} Jul 23 21:58:47.758779 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:47.758722 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" podStartSLOduration=0.758703373 podStartE2EDuration="758.703373ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:58:47.757769583 +0000 UTC m=+407.783103110" watchObservedRunningTime="2026-07-23 21:58:47.758703373 +0000 UTC m=+407.784036911" Jul 23 21:58:48.421193 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:48.421161 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:48.425662 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:48.425636 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:48.742507 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:48.742416 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:48.743321 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:48.743304 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 21:58:49.221221 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.221186 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 21:58:49.224694 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.224673 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.226959 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.226940 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4tpmk\"" Jul 23 21:58:49.227051 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.226984 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 21:58:49.227051 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.226995 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 21:58:49.227051 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.226991 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 21:58:49.234125 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.234100 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 21:58:49.276111 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.276074 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 21:58:49.373519 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.373463 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmc8\" (UniqueName: \"kubernetes.io/projected/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-kube-api-access-zhmc8\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.373682 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.373526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-config-file\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.474824 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.474738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmc8\" (UniqueName: \"kubernetes.io/projected/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-kube-api-access-zhmc8\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.474824 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.474774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-config-file\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.475460 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.475440 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-config-file\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.482868 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.482838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmc8\" (UniqueName: \"kubernetes.io/projected/5b164ffa-ab2a-40fe-bfa8-c0e37722a547-kube-api-access-zhmc8\") pod \"limitador-limitador-54d6d988d5-pxclx\" (UID: \"5b164ffa-ab2a-40fe-bfa8-c0e37722a547\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.535800 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.535766 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:49.666241 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.666216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 21:58:49.667956 ip-10-0-142-110 kubenswrapper[2569]: W0723 21:58:49.667926 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b164ffa_ab2a_40fe_bfa8_c0e37722a547.slice/crio-968125d1c4d84f63c2946a7ed251a57abe34aea1e678b6e3fe8c23b61209ce92 WatchSource:0}: Error finding container 968125d1c4d84f63c2946a7ed251a57abe34aea1e678b6e3fe8c23b61209ce92: Status 404 returned error can't find the container with id 968125d1c4d84f63c2946a7ed251a57abe34aea1e678b6e3fe8c23b61209ce92 Jul 23 21:58:49.670127 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.670110 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 21:58:49.747955 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:49.747870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" event={"ID":"5b164ffa-ab2a-40fe-bfa8-c0e37722a547","Type":"ContainerStarted","Data":"968125d1c4d84f63c2946a7ed251a57abe34aea1e678b6e3fe8c23b61209ce92"} Jul 23 21:58:52.768817 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:52.768779 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" event={"ID":"5b164ffa-ab2a-40fe-bfa8-c0e37722a547","Type":"ContainerStarted","Data":"98c1d72cce71c3d2ddb90634d745f5d55492acb3a4029511e9db270e405f2782"} Jul 23 21:58:52.769211 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:52.768905 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:58:52.787016 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:58:52.786961 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" podStartSLOduration=1.358159442 podStartE2EDuration="3.786946545s" totalImagesPullingTime="2.428787103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:58:52.785544165 +0000 UTC m=+412.810877924" watchObservedRunningTime="2026-07-23 21:58:52.786946545 +0000 UTC m=+412.812280107" Jul 23 21:59:03.776432 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:59:03.776404 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pxclx" Jul 23 21:59:59.502927 ip-10-0-142-110 kubenswrapper[2569]: I0723 21:59:59.502887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:00:07.127676 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.127587 2569 generic.go:356] "Generic (PLEG): container finished" podID="90f47074-5175-4c4b-9398-8b5f7ab490b8" containerID="96460361048dbc194b003711e1a1ae3f5851828d565c8116ed92e6141d1b203d" exitCode=0 Jul 23 22:00:07.127676 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.127661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" event={"ID":"90f47074-5175-4c4b-9398-8b5f7ab490b8","Type":"ContainerDied","Data":"96460361048dbc194b003711e1a1ae3f5851828d565c8116ed92e6141d1b203d"} Jul 23 22:00:07.128080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.128029 2569 scope.go:117] "RemoveContainer" containerID="96460361048dbc194b003711e1a1ae3f5851828d565c8116ed92e6141d1b203d" Jul 23 22:00:07.129021 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.128995 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:00:07.129095 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.129078 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:00:07.129171 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:07.129146 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:00:08.134137 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:08.134105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" event={"ID":"90f47074-5175-4c4b-9398-8b5f7ab490b8","Type":"ContainerStarted","Data":"65d027d9854eb441d21da45476c3e2d4271c0a919b8071c1b05aa727f1c546b1"} Jul 23 22:00:08.421206 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:08.421160 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 22:00:08.421424 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:08.421401 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-mltxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 23 22:00:08.421474 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:08.421451 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" podUID="90f47074-5175-4c4b-9398-8b5f7ab490b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 23 22:00:09.421155 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:09.421118 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-mltxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 23 22:00:09.421582 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:09.421178 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" podUID="90f47074-5175-4c4b-9398-8b5f7ab490b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 23 22:00:10.421591 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:10.421547 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-mltxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 23 22:00:10.421986 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:10.421613 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" podUID="90f47074-5175-4c4b-9398-8b5f7ab490b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 23 22:00:11.421632 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:11.421594 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-mltxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 23 22:00:11.422028 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:11.421651 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" podUID="90f47074-5175-4c4b-9398-8b5f7ab490b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 23 22:00:12.425206 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:12.425171 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 22:00:12.425656 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:12.425454 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 22:00:12.426136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:12.426119 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-mltxm" Jul 23 22:00:24.082511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:24.082463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:00:29.384635 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:29.384603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:00:44.191804 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:44.191704 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:00:51.287895 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:51.287858 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:00:55.685951 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:00:55.685859 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:01:05.287079 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:01:05.287041 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:02:01.669746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:02:01.669710 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:02:01.670708 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:02:01.670683 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:02:35.983028 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:02:35.982943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:02:45.188566 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:02:45.188527 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:02:56.090224 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:02:56.090182 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:04:39.290165 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:04:39.290130 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:12.485568 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:12.485533 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:20.182286 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:20.182243 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:30.080772 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:30.080689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:35.887421 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:35.887384 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:45.127824 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:45.127788 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:05:56.915746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:05:56.915712 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:06:07.586858 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:06:07.586819 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:06:17.688271 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:06:17.688236 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:06:40.286746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:06:40.286702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:06:52.688773 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:06:52.688699 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:07:01.705350 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:01.705318 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:07:01.705865 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:01.705469 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:07:16.784947 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:16.784911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:07:28.494907 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:28.494866 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:07:45.302535 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:45.302502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:07:56.797131 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:07:56.797091 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:08:06.386702 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:08:06.386653 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:08:16.111004 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:08:16.110966 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:08:24.984167 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:08:24.984087 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:10:14.891922 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:10:14.891888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:10:26.789763 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:10:26.789724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:10:35.484794 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:10:35.484749 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:10:48.182527 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:10:48.182476 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:10:57.584923 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:10:57.584881 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:11:09.688046 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:11:09.688009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:11:17.893197 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:11:17.893165 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:11:28.894557 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:11:28.894462 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:11:40.278256 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:11:40.278219 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:11:49.203939 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:11:49.203904 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:12:00.084051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:12:00.084016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:12:01.739140 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:12:01.739107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:12:01.739887 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:12:01.739869 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:12:24.194204 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:12:24.194163 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:12:31.492296 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:12:31.492255 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:13:03.984537 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:13:03.984453 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:13:19.484982 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:13:19.484943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:13:37.693299 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:13:37.693265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:13:51.683457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:13:51.683417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:13:58.584738 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:13:58.584702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:14:14.582164 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:14:14.582122 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:14:26.283070 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:14:26.282985 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:14:35.384739 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:14:35.384698 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:14:43.780708 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:14:43.780679 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:14:54.994801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:14:54.994763 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:00.146444 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.146410 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k"] Jul 23 22:15:00.149699 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.149683 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" Jul 23 22:15:00.152422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.152398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:15:00.157910 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.157879 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k"] Jul 23 22:15:00.160899 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.160879 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" Jul 23 22:15:00.292952 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.292926 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k"] Jul 23 22:15:00.294904 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:15:00.294882 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cadaf9c_8ee1_40d8_9ffb_24217c852e29.slice/crio-59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56 WatchSource:0}: Error finding container 59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56: Status 404 returned error can't find the container with id 59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56 Jul 23 22:15:00.296893 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.296875 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 22:15:00.301342 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:00.301314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" event={"ID":"0cadaf9c-8ee1-40d8-9ffb-24217c852e29","Type":"ContainerStarted","Data":"59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56"} Jul 23 22:15:03.321020 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:03.320984 2569 generic.go:356] "Generic (PLEG): container finished" podID="0cadaf9c-8ee1-40d8-9ffb-24217c852e29" containerID="f68f67c1cf2c260ca532a18943e67a13f108ce67bf20ad728579249edcd32ca9" exitCode=0 Jul 23 22:15:03.321320 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:03.321065 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" event={"ID":"0cadaf9c-8ee1-40d8-9ffb-24217c852e29","Type":"ContainerDied","Data":"f68f67c1cf2c260ca532a18943e67a13f108ce67bf20ad728579249edcd32ca9"} Jul 23 22:15:04.453729 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:04.453701 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" Jul 23 22:15:04.785707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:04.785672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:05.333644 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:05.333613 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" Jul 23 22:15:05.333644 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:05.333621 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-m4f7k" event={"ID":"0cadaf9c-8ee1-40d8-9ffb-24217c852e29","Type":"ContainerDied","Data":"59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56"} Jul 23 22:15:05.333900 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:05.333666 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59595d054d86f51cdc8517f94b9c8a178cfd353ee72bdb4977508ab94c0b4c56" Jul 23 22:15:17.488372 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:17.488337 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:26.688879 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:26.688843 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:36.588432 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:36.588394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:46.292231 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:46.292184 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:15:57.587854 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:15:57.587773 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:04.094961 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:04.094924 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:12.789218 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:12.789179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:22.103369 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:22.103338 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:34.201383 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:34.201344 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:45.619383 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:45.619348 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:16:54.527135 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:16:54.527097 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:17:01.767927 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:01.767900 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:17:01.769594 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:01.769573 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:17:16.618205 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:16.618167 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:17:23.553066 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.552981 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:17:23.553414 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.553327 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cadaf9c-8ee1-40d8-9ffb-24217c852e29" containerName="cleanup" Jul 23 22:17:23.553414 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.553338 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cadaf9c-8ee1-40d8-9ffb-24217c852e29" containerName="cleanup" Jul 23 22:17:23.553414 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.553411 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cadaf9c-8ee1-40d8-9ffb-24217c852e29" containerName="cleanup" Jul 23 22:17:23.556416 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.556388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.559931 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.559912 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d9fc566f-openshift-default-dockercfg-gjqqj\"" Jul 23 22:17:23.593648 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.593619 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:17:23.651647 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wk2s\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651721 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.651830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.652160 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.652160 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.651974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.752925 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.752873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.752925 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.752924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753183 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.752967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wk2s\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753183 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753313 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753313 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753313 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753313 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753313 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753672 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753758 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753664 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753758 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753710 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753758 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753723 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.753909 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.753878 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.755268 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.755237 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.755509 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.755472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.764228 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.764202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.764337 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.764242 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wk2s\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s\") pod \"e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:23.866082 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:23.865994 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:24.016291 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.016263 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:17:24.017326 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:17:24.017305 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbaf74bb_3134_48a6_b3c8_d675b702370d.slice/crio-46d9f9a596c11f5f009c8faae400db64255f60cb2ff23095ad7e428b57e84721 WatchSource:0}: Error finding container 46d9f9a596c11f5f009c8faae400db64255f60cb2ff23095ad7e428b57e84721: Status 404 returned error can't find the container with id 46d9f9a596c11f5f009c8faae400db64255f60cb2ff23095ad7e428b57e84721 Jul 23 22:17:24.019300 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.019265 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:24.019396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.019345 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:24.019396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.019389 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:24.999848 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.999815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerStarted","Data":"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa"} Jul 23 22:17:25.000228 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:24.999868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerStarted","Data":"46d9f9a596c11f5f009c8faae400db64255f60cb2ff23095ad7e428b57e84721"} Jul 23 22:17:25.031908 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:25.031857 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" podStartSLOduration=2.031841909 podStartE2EDuration="2.031841909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:17:25.030763004 +0000 UTC m=+1525.056096539" watchObservedRunningTime="2026-07-23 22:17:25.031841909 +0000 UTC m=+1525.057175445" Jul 23 22:17:25.866642 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:25.866610 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:25.871413 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:25.871390 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:26.004874 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:26.004841 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:26.005942 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:26.005921 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:30.293870 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:30.293836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:17:39.068464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.068428 2569 generic.go:356] "Generic (PLEG): container finished" podID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerID="5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa" exitCode=0 Jul 23 22:17:39.068942 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.068506 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerDied","Data":"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa"} Jul 23 22:17:39.068942 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.068912 2569 scope.go:117] "RemoveContainer" containerID="5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa" Jul 23 22:17:39.069884 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.069857 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:39.070080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.069924 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:39.070080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:39.069957 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:17:40.074703 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:40.074665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerStarted","Data":"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9"} Jul 23 22:17:40.867162 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:40.867125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:40.868376 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:40.868348 2569 patch_prober.go:28] interesting pod/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 23 22:17:40.868529 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:40.868402 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 23 22:17:41.005732 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:41.005684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:41.867068 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:41.867030 2569 patch_prober.go:28] interesting pod/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 23 22:17:41.867508 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:41.867090 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 23 22:17:42.866897 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:42.866858 2569 patch_prober.go:28] interesting pod/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 23 22:17:42.867082 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:42.866916 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 23 22:17:43.870201 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:43.870172 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:43.871225 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:43.871203 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:17:52.019531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.019496 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5"] Jul 23 22:17:52.022085 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.022068 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:52.024967 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.024945 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:17:52.025080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.024955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:17:52.031538 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.031511 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5"] Jul 23 22:17:52.127283 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.127235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5\" (UID: \"487522f7-886e-4221-b100-f177b809cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:52.228543 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.228500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5\" (UID: \"487522f7-886e-4221-b100-f177b809cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:52.229143 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.229124 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5\" (UID: \"487522f7-886e-4221-b100-f177b809cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:52.333734 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.333649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:52.669289 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:52.669266 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5"] Jul 23 22:17:52.671292 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:17:52.671256 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487522f7_886e_4221_b100_f177b809cdea.slice/crio-ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b WatchSource:0}: Error finding container ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b: Status 404 returned error can't find the container with id ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b Jul 23 22:17:53.139642 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:53.139608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" event={"ID":"487522f7-886e-4221-b100-f177b809cdea","Type":"ContainerStarted","Data":"718588ff87ece6ed22686abd70c53fe1f19d9b5c99d2d4c97bc1d7235856ff84"} Jul 23 22:17:53.140030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:53.139662 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" event={"ID":"487522f7-886e-4221-b100-f177b809cdea","Type":"ContainerStarted","Data":"ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b"} Jul 23 22:17:53.157932 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:53.157784 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" podStartSLOduration=1.157767112 podStartE2EDuration="1.157767112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:17:53.157743776 +0000 UTC m=+1553.183077335" watchObservedRunningTime="2026-07-23 22:17:53.157767112 +0000 UTC m=+1553.183100646" Jul 23 22:17:54.145809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:54.145774 2569 generic.go:356] "Generic (PLEG): container finished" podID="487522f7-886e-4221-b100-f177b809cdea" containerID="718588ff87ece6ed22686abd70c53fe1f19d9b5c99d2d4c97bc1d7235856ff84" exitCode=0 Jul 23 22:17:54.146203 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:54.145844 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" event={"ID":"487522f7-886e-4221-b100-f177b809cdea","Type":"ContainerDied","Data":"718588ff87ece6ed22686abd70c53fe1f19d9b5c99d2d4c97bc1d7235856ff84"} Jul 23 22:17:55.281363 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:55.281332 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:55.361766 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:55.361723 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca\") pod \"487522f7-886e-4221-b100-f177b809cdea\" (UID: \"487522f7-886e-4221-b100-f177b809cdea\") " Jul 23 22:17:55.362102 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:55.362070 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "487522f7-886e-4221-b100-f177b809cdea" (UID: "487522f7-886e-4221-b100-f177b809cdea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:17:55.463060 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:55.462967 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487522f7-886e-4221-b100-f177b809cdea-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:17:56.156595 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:56.156570 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" Jul 23 22:17:56.156756 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:56.156571 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5" event={"ID":"487522f7-886e-4221-b100-f177b809cdea","Type":"ContainerDied","Data":"ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b"} Jul 23 22:17:56.156756 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:17:56.156689 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddabcb39a773d06e55fda88aec31bf525480e8cf5bcb281b94def61afecfde7b" Jul 23 22:18:05.201449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:05.201415 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:18:07.141108 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:07.141069 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5"] Jul 23 22:18:07.142644 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:07.142624 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fc566f-zkhg5"] Jul 23 22:18:07.270019 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:07.269978 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:18:07.270280 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:07.270234 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" containerID="cri-o://08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9" gracePeriod=30 Jul 23 22:18:07.777390 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:07.777359 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487522f7-886e-4221-b100-f177b809cdea" path="/var/lib/kubelet/pods/487522f7-886e-4221-b100-f177b809cdea/volumes" Jul 23 22:18:08.371958 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.371926 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:08.372464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.372322 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487522f7-886e-4221-b100-f177b809cdea" containerName="revoke-keys" Jul 23 22:18:08.372464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.372340 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="487522f7-886e-4221-b100-f177b809cdea" containerName="revoke-keys" Jul 23 22:18:08.372464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.372429 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="487522f7-886e-4221-b100-f177b809cdea" containerName="revoke-keys" Jul 23 22:18:08.375836 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.375814 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.380816 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.380794 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c334d424-gw-openshift-default-dockercfg-hjddn\"" Jul 23 22:18:08.394202 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.394176 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:08.489139 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489308 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489308 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489308 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489377 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.489707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.489474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglh6\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590613 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglh6\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.590805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591077 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591077 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590869 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591077 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.590897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591077 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.591032 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591278 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.591134 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591612 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.591580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591756 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.591713 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.591930 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.591903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.598697 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.598667 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.598884 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.598862 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.606655 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.606631 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglh6\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.608139 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.608118 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token\") pod \"e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.687094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.687055 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:08.834343 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.834317 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:08.836616 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:18:08.836589 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c08e04_3227_4b8f_8419_2f66db774b64.slice/crio-e7ed5fe51bd1a49e673adeaaabfbab5152dc3a625e3f98ad6647449ab067e678 WatchSource:0}: Error finding container e7ed5fe51bd1a49e673adeaaabfbab5152dc3a625e3f98ad6647449ab067e678: Status 404 returned error can't find the container with id e7ed5fe51bd1a49e673adeaaabfbab5152dc3a625e3f98ad6647449ab067e678 Jul 23 22:18:08.838720 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.838685 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:08.838813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.838754 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:08.838813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:08.838785 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:09.218974 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:09.218939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerStarted","Data":"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471"} Jul 23 22:18:09.219200 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:09.219005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerStarted","Data":"e7ed5fe51bd1a49e673adeaaabfbab5152dc3a625e3f98ad6647449ab067e678"} Jul 23 22:18:09.243251 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:09.243176 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podStartSLOduration=1.243150317 podStartE2EDuration="1.243150317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:18:09.242868818 +0000 UTC m=+1569.268202380" watchObservedRunningTime="2026-07-23 22:18:09.243150317 +0000 UTC m=+1569.268483853" Jul 23 22:18:09.687882 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:09.687849 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:09.692652 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:09.692628 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:10.223879 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:10.223844 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:10.224723 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:10.224701 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:12.528601 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.528534 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:18:12.629675 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.629846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629691 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wk2s\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.629846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629726 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.629846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629745 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.629846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629782 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.629846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629822 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.630092 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629857 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.630092 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629908 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.630092 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.629972 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token\") pod \"fbaf74bb-3134-48a6-b3c8-d675b702370d\" (UID: \"fbaf74bb-3134-48a6-b3c8-d675b702370d\") " Jul 23 22:18:12.630257 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630136 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:12.630563 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630328 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.630563 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630375 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:18:12.630563 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630529 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:12.630837 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630740 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:12.630837 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.630804 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:12.632361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.632333 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:18:12.632466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.632378 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s" (OuterVolumeSpecName: "kube-api-access-4wk2s") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "kube-api-access-4wk2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:18:12.632466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.632418 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:12.632581 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.632524 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbaf74bb-3134-48a6-b3c8-d675b702370d" (UID: "fbaf74bb-3134-48a6-b3c8-d675b702370d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:18:12.731391 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731345 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731391 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731386 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731391 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731396 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731391 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731405 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wk2s\" (UniqueName: \"kubernetes.io/projected/fbaf74bb-3134-48a6-b3c8-d675b702370d-kube-api-access-4wk2s\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731703 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731420 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731703 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731431 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731703 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731442 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbaf74bb-3134-48a6-b3c8-d675b702370d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:12.731703 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:12.731450 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbaf74bb-3134-48a6-b3c8-d675b702370d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:13.238524 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.238472 2569 generic.go:356] "Generic (PLEG): container finished" podID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerID="08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9" exitCode=0 Jul 23 22:18:13.238524 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.238514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerDied","Data":"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9"} Jul 23 22:18:13.238793 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.238579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" event={"ID":"fbaf74bb-3134-48a6-b3c8-d675b702370d","Type":"ContainerDied","Data":"46d9f9a596c11f5f009c8faae400db64255f60cb2ff23095ad7e428b57e84721"} Jul 23 22:18:13.238793 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.238530 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4" Jul 23 22:18:13.238793 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.238612 2569 scope.go:117] "RemoveContainer" containerID="08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9" Jul 23 22:18:13.247922 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.247902 2569 scope.go:117] "RemoveContainer" containerID="5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa" Jul 23 22:18:13.255639 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.255617 2569 scope.go:117] "RemoveContainer" containerID="08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9" Jul 23 22:18:13.255883 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:18:13.255864 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9\": container with ID starting with 08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9 not found: ID does not exist" containerID="08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9" Jul 23 22:18:13.255940 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.255891 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9"} err="failed to get container status \"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9\": rpc error: code = NotFound desc = could not find container \"08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9\": container with ID starting with 08b748a4417e08efaf41a4f2dc1295f4fc287d9104b1d3ccf1937394d0fa23b9 not found: ID does not exist" Jul 23 22:18:13.255940 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.255907 2569 scope.go:117] "RemoveContainer" containerID="5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa" Jul 23 22:18:13.256148 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:18:13.256131 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa\": container with ID starting with 5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa not found: ID does not exist" containerID="5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa" Jul 23 22:18:13.256187 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.256155 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa"} err="failed to get container status \"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa\": rpc error: code = NotFound desc = could not find container \"5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa\": container with ID starting with 5e0dd040a27f853c9b416f2f9223fd75c09cc2d9bcd215f3f35c11ecff54b0aa not found: ID does not exist" Jul 23 22:18:13.260771 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.260751 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:18:13.264667 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.264644 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d9fc566f-openshift-default-59798fc74f-vwln4"] Jul 23 22:18:13.776766 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:13.776737 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" path="/var/lib/kubelet/pods/fbaf74bb-3134-48a6-b3c8-d675b702370d/volumes" Jul 23 22:18:18.118968 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:18.118927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:18:25.814737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:25.814182 2569 patch_prober.go:28] interesting pod/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:44258->10.134.0.34:15021: read: connection reset by peer" start-of-body= Jul 23 22:18:25.814737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:25.814265 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:44258->10.134.0.34:15021: read: connection reset by peer" Jul 23 22:18:26.303006 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.302971 2569 generic.go:356] "Generic (PLEG): container finished" podID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerID="4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471" exitCode=0 Jul 23 22:18:26.303188 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.303043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerDied","Data":"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471"} Jul 23 22:18:26.303529 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.303514 2569 scope.go:117] "RemoveContainer" containerID="4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471" Jul 23 22:18:26.304522 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.304492 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:26.304591 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.304560 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:26.304627 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:26.304591 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:27.309699 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:27.309664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerStarted","Data":"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386"} Jul 23 22:18:27.687397 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:27.687361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:27.687667 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:27.687641 2569 patch_prober.go:28] interesting pod/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 23 22:18:27.687720 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:27.687696 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 23 22:18:28.688226 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:28.688190 2569 patch_prober.go:28] interesting pod/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 23 22:18:28.688692 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:28.688250 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 23 22:18:29.687956 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:29.687920 2569 patch_prober.go:28] interesting pod/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 23 22:18:29.688146 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:29.687990 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 23 22:18:30.688422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:30.688384 2569 patch_prober.go:28] interesting pod/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 23 22:18:30.688946 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:30.688441 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 23 22:18:31.690926 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:31.690893 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:31.691377 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:31.691132 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:31.691889 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:31.691870 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:35.776728 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.776697 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f"] Jul 23 22:18:35.777101 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777057 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.777101 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777070 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.777101 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777090 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.777101 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777097 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.777246 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777172 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.777246 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.777180 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaf74bb-3134-48a6-b3c8-d675b702370d" containerName="istio-proxy" Jul 23 22:18:35.780394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.780349 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:35.783003 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.782976 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f"] Jul 23 22:18:35.783376 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.783348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:18:35.783376 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.783358 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:18:35.862330 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.862294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f\" (UID: \"0105ff9f-0062-4152-a8cb-79349ae07766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:35.963542 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.963476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f\" (UID: \"0105ff9f-0062-4152-a8cb-79349ae07766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:35.964107 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:35.964085 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f\" (UID: \"0105ff9f-0062-4152-a8cb-79349ae07766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:36.092761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:36.092674 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:36.429329 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:36.429303 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f"] Jul 23 22:18:36.431137 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:18:36.431106 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0105ff9f_0062_4152_a8cb_79349ae07766.slice/crio-487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9 WatchSource:0}: Error finding container 487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9: Status 404 returned error can't find the container with id 487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9 Jul 23 22:18:37.361609 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:37.361574 2569 generic.go:356] "Generic (PLEG): container finished" podID="0105ff9f-0062-4152-a8cb-79349ae07766" containerID="f0ab93d5d42a9761f5b46b3e429dc7d0565f250854f01830503197dfd77fb30a" exitCode=0 Jul 23 22:18:37.362055 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:37.361664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" event={"ID":"0105ff9f-0062-4152-a8cb-79349ae07766","Type":"ContainerDied","Data":"f0ab93d5d42a9761f5b46b3e429dc7d0565f250854f01830503197dfd77fb30a"} Jul 23 22:18:37.362055 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:37.361714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" event={"ID":"0105ff9f-0062-4152-a8cb-79349ae07766","Type":"ContainerStarted","Data":"487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9"} Jul 23 22:18:38.499285 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:38.499261 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:38.592181 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:38.592145 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca\") pod \"0105ff9f-0062-4152-a8cb-79349ae07766\" (UID: \"0105ff9f-0062-4152-a8cb-79349ae07766\") " Jul 23 22:18:38.592524 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:38.592502 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0105ff9f-0062-4152-a8cb-79349ae07766" (UID: "0105ff9f-0062-4152-a8cb-79349ae07766"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:18:38.693840 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:38.693807 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0105ff9f-0062-4152-a8cb-79349ae07766-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:39.373542 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:39.373509 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" Jul 23 22:18:39.373720 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:39.373513 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f" event={"ID":"0105ff9f-0062-4152-a8cb-79349ae07766","Type":"ContainerDied","Data":"487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9"} Jul 23 22:18:39.373720 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:39.373634 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487a320995e80f637090cdca6470557fef3c3653489cb993066dbf5087f494f9" Jul 23 22:18:48.988733 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:48.988695 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:18:50.852176 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:50.852139 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f"] Jul 23 22:18:50.854721 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:50.854693 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c334d424-rsc5f"] Jul 23 22:18:50.988620 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:50.988586 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:50.996965 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:50.996896 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" containerID="cri-o://2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386" gracePeriod=30 Jul 23 22:18:51.776866 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.776834 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0105ff9f-0062-4152-a8cb-79349ae07766" path="/var/lib/kubelet/pods/0105ff9f-0062-4152-a8cb-79349ae07766/volumes" Jul 23 22:18:51.822301 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.822268 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:18:51.822701 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.822686 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0105ff9f-0062-4152-a8cb-79349ae07766" containerName="revoke-keys" Jul 23 22:18:51.822752 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.822705 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0105ff9f-0062-4152-a8cb-79349ae07766" containerName="revoke-keys" Jul 23 22:18:51.822789 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.822773 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0105ff9f-0062-4152-a8cb-79349ae07766" containerName="revoke-keys" Jul 23 22:18:51.826148 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.826128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.828763 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.828739 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c965c007-openshift-default-dockercfg-zstsn\"" Jul 23 22:18:51.838079 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.838043 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:18:51.918153 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.918104 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.918781 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.918753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.918969 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.918953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919155 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919354 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqr5\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919518 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919666 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:51.919999 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:51.919872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021010 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.020915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021010 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.020977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqr5\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021229 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021265 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021354 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021468 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021713 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021625 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021747 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.021942 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.021922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.023599 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.023580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.023735 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.023716 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.029661 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.029597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.029961 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.029940 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqr5\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5\") pod \"e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.136594 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.136565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:52.276891 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.276866 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:18:52.279189 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:18:52.279151 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811fe699_cc73_4c90_91e9_de432948006b.slice/crio-664a8d7559636e2aa516d6b570bfdedc7c722c0f5cd567ee8b50d322b5bae126 WatchSource:0}: Error finding container 664a8d7559636e2aa516d6b570bfdedc7c722c0f5cd567ee8b50d322b5bae126: Status 404 returned error can't find the container with id 664a8d7559636e2aa516d6b570bfdedc7c722c0f5cd567ee8b50d322b5bae126 Jul 23 22:18:52.281416 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.281369 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:52.281493 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.281455 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:52.281550 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.281506 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:18:52.440697 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.440665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerStarted","Data":"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62"} Jul 23 22:18:52.440910 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.440722 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerStarted","Data":"664a8d7559636e2aa516d6b570bfdedc7c722c0f5cd567ee8b50d322b5bae126"} Jul 23 22:18:52.475841 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:52.475782 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podStartSLOduration=1.4757612820000001 podStartE2EDuration="1.475761282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:18:52.473693344 +0000 UTC m=+1612.499026884" watchObservedRunningTime="2026-07-23 22:18:52.475761282 +0000 UTC m=+1612.501094817" Jul 23 22:18:53.136948 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:53.136908 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:53.141571 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:53.141542 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:53.445720 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:53.445690 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:53.446570 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:53.446550 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:18:56.257650 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.257620 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:56.367110 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367034 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367110 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367094 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367125 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367149 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367198 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367238 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367269 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367300 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nglh6\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367355 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token\") pod \"b4c08e04-3227-4b8f-8419-2f66db774b64\" (UID: \"b4c08e04-3227-4b8f-8419-2f66db774b64\") " Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367454 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367511 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367625 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367678 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367680 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:18:56.367809 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.367698 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.368190 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.368165 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:56.369296 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.369265 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:18:56.369437 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.369417 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:18:56.369531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.369506 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:18:56.369706 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.369686 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6" (OuterVolumeSpecName: "kube-api-access-nglh6") pod "b4c08e04-3227-4b8f-8419-2f66db774b64" (UID: "b4c08e04-3227-4b8f-8419-2f66db774b64"). InnerVolumeSpecName "kube-api-access-nglh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:18:56.462318 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.462281 2569 generic.go:356] "Generic (PLEG): container finished" podID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerID="2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386" exitCode=0 Jul 23 22:18:56.462504 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.462334 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" Jul 23 22:18:56.462504 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.462369 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerDied","Data":"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386"} Jul 23 22:18:56.462504 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.462426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb" event={"ID":"b4c08e04-3227-4b8f-8419-2f66db774b64","Type":"ContainerDied","Data":"e7ed5fe51bd1a49e673adeaaabfbab5152dc3a625e3f98ad6647449ab067e678"} Jul 23 22:18:56.462504 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.462448 2569 scope.go:117] "RemoveContainer" containerID="2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386" Jul 23 22:18:56.468345 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468321 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468345 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468345 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c08e04-3227-4b8f-8419-2f66db774b64-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468358 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468369 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c08e04-3227-4b8f-8419-2f66db774b64-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468381 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglh6\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-kube-api-access-nglh6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468393 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.468531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.468405 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c08e04-3227-4b8f-8419-2f66db774b64-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:18:56.471614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.471592 2569 scope.go:117] "RemoveContainer" containerID="4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471" Jul 23 22:18:56.479152 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.479138 2569 scope.go:117] "RemoveContainer" containerID="2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386" Jul 23 22:18:56.479378 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:18:56.479361 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386\": container with ID starting with 2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386 not found: ID does not exist" containerID="2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386" Jul 23 22:18:56.479422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.479385 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386"} err="failed to get container status \"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386\": rpc error: code = NotFound desc = could not find container \"2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386\": container with ID starting with 2bd999a0a7d566e36e12f2cddd3d25d769f58735b3f6045d1a77509f5bccb386 not found: ID does not exist" Jul 23 22:18:56.479422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.479401 2569 scope.go:117] "RemoveContainer" containerID="4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471" Jul 23 22:18:56.479676 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:18:56.479654 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471\": container with ID starting with 4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471 not found: ID does not exist" containerID="4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471" Jul 23 22:18:56.479731 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.479682 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471"} err="failed to get container status \"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471\": rpc error: code = NotFound desc = could not find container \"4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471\": container with ID starting with 4f116d197381d4101786e93b013d8b5f88f9adfecf5df67ce53fc893ad6b1471 not found: ID does not exist" Jul 23 22:18:56.486197 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.486172 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:56.490395 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:56.490372 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c334d424-gw-openshift-default-7b99b66bd7-k4zjb"] Jul 23 22:18:57.777222 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:57.777188 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" path="/var/lib/kubelet/pods/b4c08e04-3227-4b8f-8419-2f66db774b64/volumes" Jul 23 22:18:59.192302 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:18:59.192265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:19:07.532603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.532518 2569 generic.go:356] "Generic (PLEG): container finished" podID="811fe699-cc73-4c90-91e9-de432948006b" containerID="f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62" exitCode=0 Jul 23 22:19:07.532603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.532561 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerDied","Data":"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62"} Jul 23 22:19:07.533034 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.532953 2569 scope.go:117] "RemoveContainer" containerID="f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62" Jul 23 22:19:07.533936 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.533908 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:07.533994 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.533977 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:07.534030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:07.534007 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:08.446027 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:08.445986 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:08.539782 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:08.539741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerStarted","Data":"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d"} Jul 23 22:19:09.137215 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:09.137171 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:09.138195 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:09.138172 2569 patch_prober.go:28] interesting pod/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 23 22:19:09.138303 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:09.138226 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 23 22:19:10.137502 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:10.137429 2569 patch_prober.go:28] interesting pod/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 23 22:19:10.137918 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:10.137554 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 23 22:19:11.137702 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:11.137665 2569 patch_prober.go:28] interesting pod/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 23 22:19:11.138192 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:11.137727 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 23 22:19:12.136952 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:12.136914 2569 patch_prober.go:28] interesting pod/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 23 22:19:12.137152 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:12.136979 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 23 22:19:13.141174 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:13.141142 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:13.141571 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:13.141511 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:13.142244 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:13.142225 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:32.889243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:32.889208 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:19:36.016347 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.016307 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:19:36.016777 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.016599 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" containerID="cri-o://baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d" gracePeriod=30 Jul 23 22:19:36.831177 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831140 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:19:36.831664 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831644 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:19:36.831745 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831669 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:19:36.831745 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831695 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:19:36.831745 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831701 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:19:36.831870 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.831771 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:19:36.834039 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.834018 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.837203 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.837182 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3301d6b9-openshift-default-dockercfg-5vb68\"" Jul 23 22:19:36.848607 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.848584 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:19:36.966264 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jrk\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966665 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966665 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966665 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:36.966665 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:36.966635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.067788 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jrk\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.067964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068243 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068261 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068879 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068840 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.068920 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.068899 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.070430 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.070408 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.070687 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.070652 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.076193 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.076166 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.076326 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.076303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jrk\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk\") pod \"e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.144308 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.144270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:37.282737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.282702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:19:37.285298 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:19:37.285267 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627fa75c_2808_49d5_b7fb_6105117d908d.slice/crio-6e268fe2e94ea6ff8bd3fc2dbb1e0a41560830c0891928ebfad5dd56dd4eb9c8 WatchSource:0}: Error finding container 6e268fe2e94ea6ff8bd3fc2dbb1e0a41560830c0891928ebfad5dd56dd4eb9c8: Status 404 returned error can't find the container with id 6e268fe2e94ea6ff8bd3fc2dbb1e0a41560830c0891928ebfad5dd56dd4eb9c8 Jul 23 22:19:37.287494 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.287448 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:37.287573 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.287556 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:37.287619 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.287599 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:37.681550 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.681514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerStarted","Data":"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15"} Jul 23 22:19:37.681725 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.681580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerStarted","Data":"6e268fe2e94ea6ff8bd3fc2dbb1e0a41560830c0891928ebfad5dd56dd4eb9c8"} Jul 23 22:19:37.710502 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:37.707798 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podStartSLOduration=1.707777527 podStartE2EDuration="1.707777527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:19:37.70304618 +0000 UTC m=+1657.728379729" watchObservedRunningTime="2026-07-23 22:19:37.707777527 +0000 UTC m=+1657.733111066" Jul 23 22:19:38.144719 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:38.144690 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:38.149534 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:38.149507 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:38.687069 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:38.687040 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:38.688032 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:38.688012 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:41.262588 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.262562 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:41.408722 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408685 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtqr5\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.408916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408728 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.408916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408770 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.408916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408819 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.408916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408861 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.408916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408889 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.409173 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408923 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.409173 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.408971 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.409173 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409053 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs\") pod \"811fe699-cc73-4c90-91e9-de432948006b\" (UID: \"811fe699-cc73-4c90-91e9-de432948006b\") " Jul 23 22:19:41.409173 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409082 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:19:41.409425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409395 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.409425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409403 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:19:41.409576 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409469 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:19:41.409669 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409650 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:19:41.409963 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.409938 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data" (OuterVolumeSpecName: "istio-data") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:19:41.411079 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.411058 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:19:41.411079 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.411064 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5" (OuterVolumeSpecName: "kube-api-access-wtqr5") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "kube-api-access-wtqr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:19:41.411302 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.411282 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token" (OuterVolumeSpecName: "istio-token") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:19:41.411361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.411285 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "811fe699-cc73-4c90-91e9-de432948006b" (UID: "811fe699-cc73-4c90-91e9-de432948006b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:19:41.510229 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510195 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510229 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510226 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtqr5\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-kube-api-access-wtqr5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510229 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510238 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811fe699-cc73-4c90-91e9-de432948006b-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510247 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510255 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811fe699-cc73-4c90-91e9-de432948006b-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510262 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510270 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811fe699-cc73-4c90-91e9-de432948006b-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.510457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.510277 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811fe699-cc73-4c90-91e9-de432948006b-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:19:41.704834 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.704749 2569 generic.go:356] "Generic (PLEG): container finished" podID="811fe699-cc73-4c90-91e9-de432948006b" containerID="baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d" exitCode=0 Jul 23 22:19:41.704834 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.704785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerDied","Data":"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d"} Jul 23 22:19:41.704834 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.704797 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" Jul 23 22:19:41.704834 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.704833 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9" event={"ID":"811fe699-cc73-4c90-91e9-de432948006b","Type":"ContainerDied","Data":"664a8d7559636e2aa516d6b570bfdedc7c722c0f5cd567ee8b50d322b5bae126"} Jul 23 22:19:41.705118 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.704855 2569 scope.go:117] "RemoveContainer" containerID="baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d" Jul 23 22:19:41.714009 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.713992 2569 scope.go:117] "RemoveContainer" containerID="f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62" Jul 23 22:19:41.721542 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.721521 2569 scope.go:117] "RemoveContainer" containerID="baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d" Jul 23 22:19:41.721789 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:19:41.721770 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d\": container with ID starting with baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d not found: ID does not exist" containerID="baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d" Jul 23 22:19:41.721844 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.721796 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d"} err="failed to get container status \"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d\": rpc error: code = NotFound desc = could not find container \"baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d\": container with ID starting with baf757a0524a700f5cc02ee900747a7bb43b308ec3babcca940cdd4717b6b13d not found: ID does not exist" Jul 23 22:19:41.721844 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.721812 2569 scope.go:117] "RemoveContainer" containerID="f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62" Jul 23 22:19:41.722040 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:19:41.722018 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62\": container with ID starting with f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62 not found: ID does not exist" containerID="f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62" Jul 23 22:19:41.722094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.722049 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62"} err="failed to get container status \"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62\": rpc error: code = NotFound desc = could not find container \"f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62\": container with ID starting with f6ba060eb3b06cd89a0c01eaf2a6b30927e99d5a088aceb270f0d33a0d16ce62 not found: ID does not exist" Jul 23 22:19:41.727580 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.727556 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:19:41.732670 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.732648 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c965c007-openshift-default-7dbcc9fc85-r2nx9"] Jul 23 22:19:41.778180 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:41.778144 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="811fe699-cc73-4c90-91e9-de432948006b" path="/var/lib/kubelet/pods/811fe699-cc73-4c90-91e9-de432948006b/volumes" Jul 23 22:19:45.800235 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:45.800198 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:19:53.688247 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.688212 2569 patch_prober.go:28] interesting pod/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 22:19:53.688671 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.688270 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 22:19:53.770076 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.769985 2569 generic.go:356] "Generic (PLEG): container finished" podID="627fa75c-2808-49d5-b7fb-6105117d908d" containerID="08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15" exitCode=0 Jul 23 22:19:53.770076 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.770052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerDied","Data":"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15"} Jul 23 22:19:53.770466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.770450 2569 scope.go:117] "RemoveContainer" containerID="08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15" Jul 23 22:19:53.771544 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.771515 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:53.771638 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.771584 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:53.771638 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:53.771616 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:19:54.777075 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:54.777038 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerStarted","Data":"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022"} Jul 23 22:19:55.145210 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:55.145175 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:55.145411 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:55.145388 2569 patch_prober.go:28] interesting pod/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 22:19:55.145466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:55.145448 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 22:19:56.145193 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:56.145156 2569 patch_prober.go:28] interesting pod/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 22:19:56.145697 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:56.145222 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 22:19:57.145223 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:57.145178 2569 patch_prober.go:28] interesting pod/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 22:19:57.145757 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:57.145252 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 22:19:58.145669 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:58.145633 2569 patch_prober.go:28] interesting pod/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 22:19:58.146072 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:58.145698 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 22:19:59.149220 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:59.149189 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:59.149657 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:59.149588 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:19:59.150237 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:19:59.150220 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:20:03.968777 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.968738 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8"] Jul 23 22:20:03.969172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969133 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.969172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969144 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.969172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969157 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.969172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969163 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.969324 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969231 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c08e04-3227-4b8f-8419-2f66db774b64" containerName="istio-proxy" Jul 23 22:20:03.969324 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969240 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.969324 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.969248 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="811fe699-cc73-4c90-91e9-de432948006b" containerName="istio-proxy" Jul 23 22:20:03.971465 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.971449 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:03.974159 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.974133 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:20:03.974385 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.974369 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:20:03.980614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:03.980591 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8"] Jul 23 22:20:04.026262 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.026221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8\" (UID: \"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:04.127567 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.127530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8\" (UID: \"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:04.128200 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.128182 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8\" (UID: \"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:04.283166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.283083 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:04.413261 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.413216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8"] Jul 23 22:20:04.415646 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:20:04.415618 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd7c6bd_d14c_432f_80dc_917fe2c46c5d.slice/crio-30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8 WatchSource:0}: Error finding container 30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8: Status 404 returned error can't find the container with id 30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8 Jul 23 22:20:04.834038 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.833998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" event={"ID":"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d","Type":"ContainerStarted","Data":"1a786524a028a33f053b0c09d04f57a29360c2043ff12f080d4ee11a1c80a2b5"} Jul 23 22:20:04.834242 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.834078 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" event={"ID":"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d","Type":"ContainerStarted","Data":"30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8"} Jul 23 22:20:04.850918 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:04.850866 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" podStartSLOduration=1.850852752 podStartE2EDuration="1.850852752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:20:04.849969098 +0000 UTC m=+1684.875302636" watchObservedRunningTime="2026-07-23 22:20:04.850852752 +0000 UTC m=+1684.876186280" Jul 23 22:20:05.839866 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:05.839830 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" containerID="1a786524a028a33f053b0c09d04f57a29360c2043ff12f080d4ee11a1c80a2b5" exitCode=0 Jul 23 22:20:05.839866 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:05.839869 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" event={"ID":"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d","Type":"ContainerDied","Data":"1a786524a028a33f053b0c09d04f57a29360c2043ff12f080d4ee11a1c80a2b5"} Jul 23 22:20:06.974122 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:06.974097 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:07.051107 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.051070 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca\") pod \"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d\" (UID: \"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d\") " Jul 23 22:20:07.051437 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.051412 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" (UID: "cfd7c6bd-d14c-432f-80dc-917fe2c46c5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:20:07.151980 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.151943 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:07.850123 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.850093 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" Jul 23 22:20:07.850123 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.850114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8" event={"ID":"cfd7c6bd-d14c-432f-80dc-917fe2c46c5d","Type":"ContainerDied","Data":"30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8"} Jul 23 22:20:07.850327 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:07.850154 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30ba45419df42bce7fea9bb8dffd7de0586aefa6ee851d99f99be2ba4ac67ec8" Jul 23 22:20:17.587880 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:17.587842 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:20:19.050866 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:19.050824 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8"] Jul 23 22:20:19.053106 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:19.053081 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3301d6b9-2zjr8"] Jul 23 22:20:19.188168 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:19.188130 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:20:19.188390 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:19.188347 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" containerID="cri-o://3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022" gracePeriod=30 Jul 23 22:20:19.776405 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:19.776371 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" path="/var/lib/kubelet/pods/cfd7c6bd-d14c-432f-80dc-917fe2c46c5d/volumes" Jul 23 22:20:24.442421 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.442393 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:20:24.515761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515670 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.515761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515714 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.515761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515747 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8jrk\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515782 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515849 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515885 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515915 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515956 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.515992 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:20:24.516051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516041 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy\") pod \"627fa75c-2808-49d5-b7fb-6105117d908d\" (UID: \"627fa75c-2808-49d5-b7fb-6105117d908d\") " Jul 23 22:20:24.516449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516373 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.516449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516034 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:20:24.516449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516433 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:20:24.516652 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516472 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:20:24.517009 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.516984 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data" (OuterVolumeSpecName: "istio-data") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:20:24.518099 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.518074 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token" (OuterVolumeSpecName: "istio-token") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:20:24.518219 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.518097 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:20:24.518404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.518386 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:20:24.518459 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.518399 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk" (OuterVolumeSpecName: "kube-api-access-b8jrk") pod "627fa75c-2808-49d5-b7fb-6105117d908d" (UID: "627fa75c-2808-49d5-b7fb-6105117d908d"). InnerVolumeSpecName "kube-api-access-b8jrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:20:24.617425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617378 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617423 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627fa75c-2808-49d5-b7fb-6105117d908d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617433 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617425 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617442 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627fa75c-2808-49d5-b7fb-6105117d908d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617729 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617450 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617729 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617457 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617729 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617466 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627fa75c-2808-49d5-b7fb-6105117d908d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.617729 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.617473 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jrk\" (UniqueName: \"kubernetes.io/projected/627fa75c-2808-49d5-b7fb-6105117d908d-kube-api-access-b8jrk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:20:24.937544 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.937506 2569 generic.go:356] "Generic (PLEG): container finished" podID="627fa75c-2808-49d5-b7fb-6105117d908d" containerID="3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022" exitCode=0 Jul 23 22:20:24.937746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.937560 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" Jul 23 22:20:24.937746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.937572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerDied","Data":"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022"} Jul 23 22:20:24.937746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.937622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p" event={"ID":"627fa75c-2808-49d5-b7fb-6105117d908d","Type":"ContainerDied","Data":"6e268fe2e94ea6ff8bd3fc2dbb1e0a41560830c0891928ebfad5dd56dd4eb9c8"} Jul 23 22:20:24.937746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.937645 2569 scope.go:117] "RemoveContainer" containerID="3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022" Jul 23 22:20:24.947373 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.947353 2569 scope.go:117] "RemoveContainer" containerID="08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15" Jul 23 22:20:24.955030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.955010 2569 scope.go:117] "RemoveContainer" containerID="3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022" Jul 23 22:20:24.955279 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:20:24.955262 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022\": container with ID starting with 3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022 not found: ID does not exist" containerID="3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022" Jul 23 22:20:24.955327 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.955286 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022"} err="failed to get container status \"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022\": rpc error: code = NotFound desc = could not find container \"3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022\": container with ID starting with 3939e30ec4c1c828d9670b8331d81844f7d85b3b406519dabe176d47443fb022 not found: ID does not exist" Jul 23 22:20:24.955327 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.955302 2569 scope.go:117] "RemoveContainer" containerID="08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15" Jul 23 22:20:24.955595 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:20:24.955575 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15\": container with ID starting with 08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15 not found: ID does not exist" containerID="08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15" Jul 23 22:20:24.955644 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.955602 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15"} err="failed to get container status \"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15\": rpc error: code = NotFound desc = could not find container \"08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15\": container with ID starting with 08b038cacfd2a6c75dc0faef82bfa508bd793e09806902d0dbb4f03c3633af15 not found: ID does not exist" Jul 23 22:20:24.959869 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.959847 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:20:24.963520 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:24.963474 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3301d6b9-openshift-default-7db58c5bdc-9628p"] Jul 23 22:20:25.777191 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:20:25.777155 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" path="/var/lib/kubelet/pods/627fa75c-2808-49d5-b7fb-6105117d908d/volumes" Jul 23 22:21:46.378647 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:46.378612 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:21:57.007539 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.007442 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:21:57.299970 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.299884 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:21:57.300302 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300286 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:21:57.300345 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300304 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:21:57.300345 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300321 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" containerName="revoke-keys" Jul 23 22:21:57.300345 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300327 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" containerName="revoke-keys" Jul 23 22:21:57.300440 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300403 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:21:57.300440 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300412 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:21:57.300440 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.300419 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd7c6bd-d14c-432f-80dc-917fe2c46c5d" containerName="revoke-keys" Jul 23 22:21:57.303730 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.303710 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.307493 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.307456 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5d321d11-openshift-default-dockercfg-s94qv\"" Jul 23 22:21:57.328698 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.328663 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:21:57.500860 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.500822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.500880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.500924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501051 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.500943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhpv9\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.501048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.501090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.501113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.501133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.501166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.501152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602078 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.601984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602078 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602078 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhpv9\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602357 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602705 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602442 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602705 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602705 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.602705 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602677 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.603016 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.602997 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.604669 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.604645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.604793 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.604680 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.610766 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.610736 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.610854 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.610771 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhpv9\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9\") pod \"e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.614612 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.614592 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:57.766558 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.766520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:21:57.768935 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:21:57.768902 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a8e0a9_2bf9_43c6_8137_28e38937b73e.slice/crio-21e7a5208d89a25e5887c193efee5f35ff01a7bfd3d58b5ffd2a5e3bcce92d4b WatchSource:0}: Error finding container 21e7a5208d89a25e5887c193efee5f35ff01a7bfd3d58b5ffd2a5e3bcce92d4b: Status 404 returned error can't find the container with id 21e7a5208d89a25e5887c193efee5f35ff01a7bfd3d58b5ffd2a5e3bcce92d4b Jul 23 22:21:57.771632 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.771581 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:21:57.771743 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.771685 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:21:57.771788 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:57.771739 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:21:58.411635 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:58.411598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerStarted","Data":"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6"} Jul 23 22:21:58.412128 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:58.411658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerStarted","Data":"21e7a5208d89a25e5887c193efee5f35ff01a7bfd3d58b5ffd2a5e3bcce92d4b"} Jul 23 22:21:58.434737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:58.434677 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podStartSLOduration=1.434659412 podStartE2EDuration="1.434659412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:21:58.433240413 +0000 UTC m=+1798.458573948" watchObservedRunningTime="2026-07-23 22:21:58.434659412 +0000 UTC m=+1798.459992948" Jul 23 22:21:58.615502 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:58.615442 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:58.620296 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:58.620272 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:59.416371 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:59.416342 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:21:59.417272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:21:59.417253 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:01.810663 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:01.810629 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:22:01.814548 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:01.814525 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:22:06.596421 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:06.596386 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:22:12.448445 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.448414 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8"] Jul 23 22:22:12.448882 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.448805 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:22:12.448882 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.448818 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="627fa75c-2808-49d5-b7fb-6105117d908d" containerName="istio-proxy" Jul 23 22:22:12.451894 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.451876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:12.454881 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.454857 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:22:12.455008 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.454865 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:22:12.461988 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.461961 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8"] Jul 23 22:22:12.551306 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.551264 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8\" (UID: \"48e44b9f-7d27-4639-98ca-06c0165e9486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:12.652424 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.652386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8\" (UID: \"48e44b9f-7d27-4639-98ca-06c0165e9486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:12.653294 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.653267 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8\" (UID: \"48e44b9f-7d27-4639-98ca-06c0165e9486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:12.763976 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.763893 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:12.902171 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:12.902143 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8"] Jul 23 22:22:12.904085 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:22:12.904053 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e44b9f_7d27_4639_98ca_06c0165e9486.slice/crio-3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf WatchSource:0}: Error finding container 3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf: Status 404 returned error can't find the container with id 3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf Jul 23 22:22:13.493337 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:13.493303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" event={"ID":"48e44b9f-7d27-4639-98ca-06c0165e9486","Type":"ContainerStarted","Data":"7e0841c3fb0dfc7b454674d135c6719deb318d8c55215a59afa8990fad31ceb5"} Jul 23 22:22:13.493808 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:13.493352 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" event={"ID":"48e44b9f-7d27-4639-98ca-06c0165e9486","Type":"ContainerStarted","Data":"3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf"} Jul 23 22:22:13.508750 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:13.508706 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" podStartSLOduration=1.508687425 podStartE2EDuration="1.508687425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:13.508319725 +0000 UTC m=+1813.533653267" watchObservedRunningTime="2026-07-23 22:22:13.508687425 +0000 UTC m=+1813.534020962" Jul 23 22:22:14.417581 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.417545 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 22:22:14.417787 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.417603 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 22:22:14.500683 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.500651 2569 generic.go:356] "Generic (PLEG): container finished" podID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerID="6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6" exitCode=0 Jul 23 22:22:14.501155 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.500735 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerDied","Data":"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6"} Jul 23 22:22:14.501232 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.501196 2569 scope.go:117] "RemoveContainer" containerID="6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6" Jul 23 22:22:14.502255 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.502223 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:14.502360 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.502295 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:14.502360 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.502317 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" event={"ID":"48e44b9f-7d27-4639-98ca-06c0165e9486","Type":"ContainerDied","Data":"7e0841c3fb0dfc7b454674d135c6719deb318d8c55215a59afa8990fad31ceb5"} Jul 23 22:22:14.502360 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.502335 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:14.502496 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:14.502295 2569 generic.go:356] "Generic (PLEG): container finished" podID="48e44b9f-7d27-4639-98ca-06c0165e9486" containerID="7e0841c3fb0dfc7b454674d135c6719deb318d8c55215a59afa8990fad31ceb5" exitCode=0 Jul 23 22:22:15.514289 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.514247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerStarted","Data":"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc"} Jul 23 22:22:15.616564 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.616207 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:15.616564 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.616335 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 22:22:15.616564 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.616387 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 22:22:15.669794 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.669768 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:15.786775 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.786692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca\") pod \"48e44b9f-7d27-4639-98ca-06c0165e9486\" (UID: \"48e44b9f-7d27-4639-98ca-06c0165e9486\") " Jul 23 22:22:15.786991 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.786968 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48e44b9f-7d27-4639-98ca-06c0165e9486" (UID: "48e44b9f-7d27-4639-98ca-06c0165e9486"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:22:15.888208 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:15.888174 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e44b9f-7d27-4639-98ca-06c0165e9486-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:16.520690 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:16.520658 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" Jul 23 22:22:16.521097 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:16.520711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8" event={"ID":"48e44b9f-7d27-4639-98ca-06c0165e9486","Type":"ContainerDied","Data":"3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf"} Jul 23 22:22:16.521097 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:16.520752 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3de2463c46362ea5db75aad19e839203c8b45ff63cffcde606281876112edf" Jul 23 22:22:16.615557 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:16.615517 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 22:22:16.615727 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:16.615586 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 22:22:17.615958 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:17.615918 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 22:22:17.616553 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:17.615989 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 22:22:18.615603 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:18.615565 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 22:22:18.615826 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:18.615632 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 22:22:19.619805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:19.619775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:19.620229 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:19.620157 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:19.620857 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:19.620839 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:25.688471 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:25.688434 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:22:27.534418 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:27.534382 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8"] Jul 23 22:22:27.536757 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:27.536731 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d321d11-qk9j8"] Jul 23 22:22:27.776736 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:27.776705 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e44b9f-7d27-4639-98ca-06c0165e9486" path="/var/lib/kubelet/pods/48e44b9f-7d27-4639-98ca-06c0165e9486/volumes" Jul 23 22:22:34.179823 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:34.179785 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:22:34.180335 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:34.180079 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" containerID="cri-o://48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc" gracePeriod=30 Jul 23 22:22:34.622140 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:34.622056 2569 patch_prober.go:28] interesting pod/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 23 22:22:34.622140 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:34.622112 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 22:22:39.427118 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.427086 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:39.523804 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523715 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.523804 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523747 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.523804 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523769 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.523804 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523801 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523838 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523893 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523920 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.523982 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524058 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:22:39.524136 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524113 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhpv9\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9\") pod \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\" (UID: \"67a8e0a9-2bf9-43c6-8137-28e38937b73e\") " Jul 23 22:22:39.524458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524201 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:22:39.524458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524346 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:22:39.524625 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524599 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.524688 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524630 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.524688 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524634 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:22:39.524688 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524649 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.524688 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.524671 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data" (OuterVolumeSpecName: "istio-data") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:22:39.526350 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.526318 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9" (OuterVolumeSpecName: "kube-api-access-zhpv9") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "kube-api-access-zhpv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:22:39.526350 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.526325 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:22:39.526350 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.526327 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token" (OuterVolumeSpecName: "istio-token") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:22:39.526551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.526373 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67a8e0a9-2bf9-43c6-8137-28e38937b73e" (UID: "67a8e0a9-2bf9-43c6-8137-28e38937b73e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:22:39.625466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625428 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.625466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625462 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.625740 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625472 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhpv9\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-kube-api-access-zhpv9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.625740 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625507 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.625740 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625515 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.625740 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.625522 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a8e0a9-2bf9-43c6-8137-28e38937b73e-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:22:39.646711 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.646672 2569 generic.go:356] "Generic (PLEG): container finished" podID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerID="48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc" exitCode=0 Jul 23 22:22:39.646864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.646731 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" Jul 23 22:22:39.646864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.646748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerDied","Data":"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc"} Jul 23 22:22:39.646864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.646812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd" event={"ID":"67a8e0a9-2bf9-43c6-8137-28e38937b73e","Type":"ContainerDied","Data":"21e7a5208d89a25e5887c193efee5f35ff01a7bfd3d58b5ffd2a5e3bcce92d4b"} Jul 23 22:22:39.646864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.646834 2569 scope.go:117] "RemoveContainer" containerID="48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc" Jul 23 22:22:39.656149 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.656128 2569 scope.go:117] "RemoveContainer" containerID="6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6" Jul 23 22:22:39.664046 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.664022 2569 scope.go:117] "RemoveContainer" containerID="48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc" Jul 23 22:22:39.664313 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:22:39.664291 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc\": container with ID starting with 48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc not found: ID does not exist" containerID="48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc" Jul 23 22:22:39.664394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.664320 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc"} err="failed to get container status \"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc\": rpc error: code = NotFound desc = could not find container \"48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc\": container with ID starting with 48ba2fd16de6da6f9a64050182f85669303698de087f5f0fc40ccf56ae7569bc not found: ID does not exist" Jul 23 22:22:39.664394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.664337 2569 scope.go:117] "RemoveContainer" containerID="6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6" Jul 23 22:22:39.664612 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:22:39.664597 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6\": container with ID starting with 6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6 not found: ID does not exist" containerID="6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6" Jul 23 22:22:39.664664 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.664617 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6"} err="failed to get container status \"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6\": rpc error: code = NotFound desc = could not find container \"6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6\": container with ID starting with 6ac4c01b0e68f922d5195f1a1a876ecaed19b7a83b48e844380db597c826a7e6 not found: ID does not exist" Jul 23 22:22:39.669627 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.669604 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:22:39.673678 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.673656 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5d321d11-openshift-default-7c7f6479dc-dsxpd"] Jul 23 22:22:39.777830 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:39.777747 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" path="/var/lib/kubelet/pods/67a8e0a9-2bf9-43c6-8137-28e38937b73e/volumes" Jul 23 22:22:53.849822 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.849781 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:22:53.850264 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850210 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:22:53.850264 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850225 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:22:53.850264 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850238 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e44b9f-7d27-4639-98ca-06c0165e9486" containerName="revoke-keys" Jul 23 22:22:53.850264 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850246 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e44b9f-7d27-4639-98ca-06c0165e9486" containerName="revoke-keys" Jul 23 22:22:53.850397 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850330 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e44b9f-7d27-4639-98ca-06c0165e9486" containerName="revoke-keys" Jul 23 22:22:53.850397 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850341 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:22:53.850397 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.850348 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:22:53.855475 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.855459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.857978 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.857956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0578d301-openshift-default-dockercfg-lrx5c\"" Jul 23 22:22:53.866933 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.866899 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:22:53.962326 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962725 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962725 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962725 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4n8\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:53.962725 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:53.962678 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064201 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064209 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4n8\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064709 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064709 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064709 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064709 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064658 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064902 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.064961 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.064895 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.065080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.065057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.065185 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.065078 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.066634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.066614 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.066934 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.066909 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.072815 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.072794 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.072899 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.072867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4n8\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8\") pod \"e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.166188 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.166153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:54.299003 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.298980 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:22:54.300759 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:22:54.300731 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9671cb_6306_4476_ae00_8058326dbba5.slice/crio-ac81ebb4ae49770fdf23eee81b343ed5e1aefc4dadad5b043b9975a80529cacf WatchSource:0}: Error finding container ac81ebb4ae49770fdf23eee81b343ed5e1aefc4dadad5b043b9975a80529cacf: Status 404 returned error can't find the container with id ac81ebb4ae49770fdf23eee81b343ed5e1aefc4dadad5b043b9975a80529cacf Jul 23 22:22:54.302705 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.302667 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:54.302807 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.302731 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:54.302807 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.302760 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:22:54.723129 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.723097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerStarted","Data":"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4"} Jul 23 22:22:54.723330 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.723158 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerStarted","Data":"ac81ebb4ae49770fdf23eee81b343ed5e1aefc4dadad5b043b9975a80529cacf"} Jul 23 22:22:54.747993 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:54.747936 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podStartSLOduration=1.747921155 podStartE2EDuration="1.747921155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:54.746554997 +0000 UTC m=+1854.771888532" watchObservedRunningTime="2026-07-23 22:22:54.747921155 +0000 UTC m=+1854.773254690" Jul 23 22:22:55.166513 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:55.166459 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:55.171350 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:55.171324 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:55.727745 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:55.727710 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:22:55.728624 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:22:55.728606 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:01.827980 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:01.827953 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:23:09.804268 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.804192 2569 generic.go:356] "Generic (PLEG): container finished" podID="9b9671cb-6306-4476-ae00-8058326dbba5" containerID="919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4" exitCode=0 Jul 23 22:23:09.804691 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.804263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerDied","Data":"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4"} Jul 23 22:23:09.804732 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.804693 2569 scope.go:117] "RemoveContainer" containerID="919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4" Jul 23 22:23:09.805748 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.805721 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:23:09.805816 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.805791 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:23:09.805852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:09.805819 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:23:10.727833 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:10.727795 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:10.815991 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:10.815952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerStarted","Data":"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50"} Jul 23 22:23:11.166779 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:11.166744 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:11.167894 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:11.167868 2569 patch_prober.go:28] interesting pod/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 23 22:23:11.168004 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:11.167929 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 23 22:23:12.166794 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:12.166752 2569 patch_prober.go:28] interesting pod/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 23 22:23:12.167276 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:12.166824 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 23 22:23:13.166659 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:13.166620 2569 patch_prober.go:28] interesting pod/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 23 22:23:13.166846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:13.166686 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 23 22:23:14.166667 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:14.166625 2569 patch_prober.go:28] interesting pod/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 23 22:23:14.166854 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:14.166686 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 23 22:23:15.171045 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:15.171008 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:15.171564 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:15.171402 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:15.172123 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:15.172105 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:23:19.400561 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:19.400519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:23:38.896474 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:38.896423 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:23:45.074513 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.074462 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw"] Jul 23 22:23:45.075034 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.075013 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:23:45.075112 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.075039 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a8e0a9-2bf9-43c6-8137-28e38937b73e" containerName="istio-proxy" Jul 23 22:23:45.078212 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.078191 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:45.080797 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.080774 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:23:45.080908 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.080826 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:23:45.085373 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.085346 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw"] Jul 23 22:23:45.256893 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.256856 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0578d301-gpcxw\" (UID: \"60fb8238-8da4-47ae-8323-46bcb7e9ffd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:45.357958 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.357873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0578d301-gpcxw\" (UID: \"60fb8238-8da4-47ae-8323-46bcb7e9ffd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:45.358498 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.358466 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0578d301-gpcxw\" (UID: \"60fb8238-8da4-47ae-8323-46bcb7e9ffd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:45.389873 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.389842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:45.517803 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.517772 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw"] Jul 23 22:23:45.520698 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:23:45.520670 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fb8238_8da4_47ae_8323_46bcb7e9ffd9.slice/crio-45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173 WatchSource:0}: Error finding container 45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173: Status 404 returned error can't find the container with id 45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173 Jul 23 22:23:45.996199 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.996107 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" event={"ID":"60fb8238-8da4-47ae-8323-46bcb7e9ffd9","Type":"ContainerStarted","Data":"61c51f2bf0b8b7417728dde834fac51ef9a0509d83aaa196d8b60833e1c72973"} Jul 23 22:23:45.996199 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:45.996184 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" event={"ID":"60fb8238-8da4-47ae-8323-46bcb7e9ffd9","Type":"ContainerStarted","Data":"45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173"} Jul 23 22:23:46.017304 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:46.017240 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" podStartSLOduration=1.017221381 podStartE2EDuration="1.017221381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:46.016803178 +0000 UTC m=+1906.042136712" watchObservedRunningTime="2026-07-23 22:23:46.017221381 +0000 UTC m=+1906.042554914" Jul 23 22:23:47.004192 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:47.004161 2569 generic.go:356] "Generic (PLEG): container finished" podID="60fb8238-8da4-47ae-8323-46bcb7e9ffd9" containerID="61c51f2bf0b8b7417728dde834fac51ef9a0509d83aaa196d8b60833e1c72973" exitCode=0 Jul 23 22:23:47.004576 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:47.004248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" event={"ID":"60fb8238-8da4-47ae-8323-46bcb7e9ffd9","Type":"ContainerDied","Data":"61c51f2bf0b8b7417728dde834fac51ef9a0509d83aaa196d8b60833e1c72973"} Jul 23 22:23:48.139854 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:48.139830 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:48.288529 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:48.288404 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca\") pod \"60fb8238-8da4-47ae-8323-46bcb7e9ffd9\" (UID: \"60fb8238-8da4-47ae-8323-46bcb7e9ffd9\") " Jul 23 22:23:48.288855 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:48.288829 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60fb8238-8da4-47ae-8323-46bcb7e9ffd9" (UID: "60fb8238-8da4-47ae-8323-46bcb7e9ffd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:23:48.389219 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:48.389179 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fb8238-8da4-47ae-8323-46bcb7e9ffd9-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:23:49.016157 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:49.016130 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" Jul 23 22:23:49.016378 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:49.016172 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw" event={"ID":"60fb8238-8da4-47ae-8323-46bcb7e9ffd9","Type":"ContainerDied","Data":"45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173"} Jul 23 22:23:49.016378 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:49.016222 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45eae6ccdf004d405273db569e04b69c26f7572ee20990f8a0b7abf2d426e173" Jul 23 22:23:59.289304 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:23:59.289270 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:24:01.877408 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:24:01.877374 2569 scope.go:117] "RemoveContainer" containerID="718588ff87ece6ed22686abd70c53fe1f19d9b5c99d2d4c97bc1d7235856ff84" Jul 23 22:24:05.185270 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:24:05.185229 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw"] Jul 23 22:24:05.187127 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:24:05.187104 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0578d301-gpcxw"] Jul 23 22:24:05.777097 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:24:05.777057 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60fb8238-8da4-47ae-8323-46bcb7e9ffd9" path="/var/lib/kubelet/pods/60fb8238-8da4-47ae-8323-46bcb7e9ffd9/volumes" Jul 23 22:24:09.686773 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:24:09.686730 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:25:01.899609 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:01.899533 2569 scope.go:117] "RemoveContainer" containerID="f0ab93d5d42a9761f5b46b3e429dc7d0565f250854f01830503197dfd77fb30a" Jul 23 22:25:04.989903 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:04.989872 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:25:04.990349 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:04.990071 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" containerID="cri-o://6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50" gracePeriod=30 Jul 23 22:25:10.241846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.241816 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:25:10.269222 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269140 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz4n8\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269222 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269182 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269222 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269214 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269250 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269298 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269316 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269347 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket\") pod \"9b9671cb-6306-4476-ae00-8058326dbba5\" (UID: \"9b9671cb-6306-4476-ae00-8058326dbba5\") " Jul 23 22:25:10.269836 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269643 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:25:10.269836 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269765 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.269944 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269861 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:25:10.270005 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.269951 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:25:10.270063 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.270016 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:25:10.270417 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.270389 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:25:10.271560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.271535 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:25:10.271659 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.271580 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:25:10.271659 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.271623 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8" (OuterVolumeSpecName: "kube-api-access-bz4n8") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "kube-api-access-bz4n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:25:10.271736 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.271653 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b9671cb-6306-4476-ae00-8058326dbba5" (UID: "9b9671cb-6306-4476-ae00-8058326dbba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:25:10.370384 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370343 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4n8\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-kube-api-access-bz4n8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370384 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370374 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370384 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370385 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9671cb-6306-4476-ae00-8058326dbba5-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370384 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370395 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370694 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370403 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370694 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370411 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9671cb-6306-4476-ae00-8058326dbba5-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370694 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370418 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9671cb-6306-4476-ae00-8058326dbba5-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.370694 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.370425 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9671cb-6306-4476-ae00-8058326dbba5-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:25:10.418918 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.418887 2569 generic.go:356] "Generic (PLEG): container finished" podID="9b9671cb-6306-4476-ae00-8058326dbba5" containerID="6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50" exitCode=0 Jul 23 22:25:10.419094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.418932 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" Jul 23 22:25:10.419094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.418979 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerDied","Data":"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50"} Jul 23 22:25:10.419094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.419038 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h" event={"ID":"9b9671cb-6306-4476-ae00-8058326dbba5","Type":"ContainerDied","Data":"ac81ebb4ae49770fdf23eee81b343ed5e1aefc4dadad5b043b9975a80529cacf"} Jul 23 22:25:10.419094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.419067 2569 scope.go:117] "RemoveContainer" containerID="6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50" Jul 23 22:25:10.428143 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.428125 2569 scope.go:117] "RemoveContainer" containerID="919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4" Jul 23 22:25:10.435832 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.435815 2569 scope.go:117] "RemoveContainer" containerID="6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50" Jul 23 22:25:10.436080 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:25:10.436061 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50\": container with ID starting with 6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50 not found: ID does not exist" containerID="6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50" Jul 23 22:25:10.436152 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.436093 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50"} err="failed to get container status \"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50\": rpc error: code = NotFound desc = could not find container \"6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50\": container with ID starting with 6b3aad765f604f25787386460f779b11f5fddff1d0c43c13a46c45557c224b50 not found: ID does not exist" Jul 23 22:25:10.436152 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.436118 2569 scope.go:117] "RemoveContainer" containerID="919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4" Jul 23 22:25:10.436361 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:25:10.436343 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4\": container with ID starting with 919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4 not found: ID does not exist" containerID="919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4" Jul 23 22:25:10.436397 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.436369 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4"} err="failed to get container status \"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4\": rpc error: code = NotFound desc = could not find container \"919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4\": container with ID starting with 919e82bb6795edea089c7d69d72e9ef5ec94d3b470afcdcb9e2409a0b0f319e4 not found: ID does not exist" Jul 23 22:25:10.457923 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.457889 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:25:10.473275 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:10.473242 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0578d301-openshift-default-5bcc56d95f-w2c2h"] Jul 23 22:25:11.777201 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:11.777158 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" path="/var/lib/kubelet/pods/9b9671cb-6306-4476-ae00-8058326dbba5/volumes" Jul 23 22:25:15.793121 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:15.793084 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:25:23.293078 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:25:23.293039 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:26:01.220129 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.220088 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:26:01.221317 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221292 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.221456 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221444 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.221585 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221573 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60fb8238-8da4-47ae-8323-46bcb7e9ffd9" containerName="revoke-keys" Jul 23 22:26:01.221666 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221657 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60fb8238-8da4-47ae-8323-46bcb7e9ffd9" containerName="revoke-keys" Jul 23 22:26:01.221737 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221728 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.221806 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221798 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.221974 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.221964 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.222054 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.222045 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9671cb-6306-4476-ae00-8058326dbba5" containerName="istio-proxy" Jul 23 22:26:01.222121 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.222112 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60fb8238-8da4-47ae-8323-46bcb7e9ffd9" containerName="revoke-keys" Jul 23 22:26:01.225896 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.225870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.229257 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.229235 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-071748-openshift-default-dockercfg-wcdpq\"" Jul 23 22:26:01.243774 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.243745 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:26:01.356836 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.356789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdktz\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.356857 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.356874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.356903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.356930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.357007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357047 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.357041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357278 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.357086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.357278 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.357120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.458666 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.458867 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.458867 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdktz\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.458867 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459029 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459029 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.458990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459138 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459138 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459138 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459309 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459252 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459440 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459418 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459527 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459470 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459598 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459562 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.459655 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.459622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.461233 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.461206 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.461354 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.461336 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.469979 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.469953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.470126 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.470091 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdktz\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz\") pod \"e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.536748 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.536661 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:01.689850 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.689820 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:26:01.691967 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:26:01.691936 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87484ca5_0015_4450_9115_88eeda518f87.slice/crio-1fd96abd28af7351b600e712a026bc4bdd5046a791626c212de8c69fb23ad87d WatchSource:0}: Error finding container 1fd96abd28af7351b600e712a026bc4bdd5046a791626c212de8c69fb23ad87d: Status 404 returned error can't find the container with id 1fd96abd28af7351b600e712a026bc4bdd5046a791626c212de8c69fb23ad87d Jul 23 22:26:01.694129 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.694092 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:01.694210 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.694161 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:01.694249 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:01.694212 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:02.668564 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:02.668527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerStarted","Data":"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058"} Jul 23 22:26:02.668934 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:02.668616 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerStarted","Data":"1fd96abd28af7351b600e712a026bc4bdd5046a791626c212de8c69fb23ad87d"} Jul 23 22:26:02.695648 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:02.695587 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podStartSLOduration=1.6955714450000001 podStartE2EDuration="1.695571445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:26:02.693256062 +0000 UTC m=+2042.718589601" watchObservedRunningTime="2026-07-23 22:26:02.695571445 +0000 UTC m=+2042.720904958" Jul 23 22:26:03.537109 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:03.537060 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:03.542050 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:03.542024 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:03.672987 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:03.672953 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:03.673811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:03.673793 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:09.899775 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:09.899737 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:26:09.903088 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:09.903071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:09.908802 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:09.908773 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b12a8c-openshift-default-dockercfg-jdvtr\"" Jul 23 22:26:09.934947 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:09.934916 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:26:09.963332 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:09.963301 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:26:10.049407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nctn\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049750 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049750 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049619 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049750 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049750 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049885 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049885 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049863 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.049949 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.049909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150623 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150623 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150623 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nctn\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150927 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150646 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150927 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150688 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.150927 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150723 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.150961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151007 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151317 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151110 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151365 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.151739 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.151720 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.153034 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.153015 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.153131 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.153101 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.159916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.159889 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.160065 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.160046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nctn\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn\") pod \"e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.215181 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.215030 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:10.366742 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.366710 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:26:10.368819 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:26:10.368791 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14fa5d4_e3db_45a7_b039_faa01b45432d.slice/crio-4a6ff57f0005804195edcb31397479563f9e06c1f1c9a6ffbee2813a0ce4150c WatchSource:0}: Error finding container 4a6ff57f0005804195edcb31397479563f9e06c1f1c9a6ffbee2813a0ce4150c: Status 404 returned error can't find the container with id 4a6ff57f0005804195edcb31397479563f9e06c1f1c9a6ffbee2813a0ce4150c Jul 23 22:26:10.371346 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.371314 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:10.371423 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.371381 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:10.371423 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.371410 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:10.709947 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.709857 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerStarted","Data":"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c"} Jul 23 22:26:10.709947 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.709907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerStarted","Data":"4a6ff57f0005804195edcb31397479563f9e06c1f1c9a6ffbee2813a0ce4150c"} Jul 23 22:26:10.731733 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:10.731672 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podStartSLOduration=1.731656254 podStartE2EDuration="1.731656254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:26:10.729919752 +0000 UTC m=+2050.755253289" watchObservedRunningTime="2026-07-23 22:26:10.731656254 +0000 UTC m=+2050.756989789" Jul 23 22:26:11.216248 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:11.216200 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:11.221799 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:11.221771 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:11.716352 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:11.716312 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:11.717463 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:11.717438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:17.088821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:17.088776 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:26:18.674036 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.674000 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 22:26:18.674457 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.674065 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 22:26:18.755004 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.754971 2569 generic.go:356] "Generic (PLEG): container finished" podID="87484ca5-0015-4450-9115-88eeda518f87" containerID="e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058" exitCode=0 Jul 23 22:26:18.755184 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.755042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerDied","Data":"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058"} Jul 23 22:26:18.755466 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.755452 2569 scope.go:117] "RemoveContainer" containerID="e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058" Jul 23 22:26:18.756468 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.756440 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:18.756542 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.756526 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:18.756579 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:18.756558 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:19.762192 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:19.762161 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerStarted","Data":"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70"} Jul 23 22:26:20.537116 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:20.537069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:20.537893 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:20.537845 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 22:26:20.538046 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:20.537926 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 22:26:21.537941 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:21.537890 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 22:26:21.538464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:21.537996 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 22:26:22.537424 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:22.537319 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 22:26:22.537424 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:22.537392 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 22:26:23.537570 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:23.537530 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 22:26:23.537967 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:23.537613 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 22:26:24.541768 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:24.541735 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:24.542262 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:24.542117 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:24.542915 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:24.542896 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:26:25.798594 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.798563 2569 generic.go:356] "Generic (PLEG): container finished" podID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerID="270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c" exitCode=0 Jul 23 22:26:25.799037 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.798645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerDied","Data":"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c"} Jul 23 22:26:25.799186 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.799170 2569 scope.go:117] "RemoveContainer" containerID="270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c" Jul 23 22:26:25.800166 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.800137 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:25.800236 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.800219 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:25.800285 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:25.800269 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:26:26.716394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:26.716343 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:26.811154 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:26.811117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerStarted","Data":"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595"} Jul 23 22:26:27.215733 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:27.215693 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:27.217066 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:27.217042 2569 patch_prober.go:28] interesting pod/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 22:26:27.217175 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:27.217093 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 22:26:28.216215 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:28.216167 2569 patch_prober.go:28] interesting pod/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 22:26:28.216753 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:28.216223 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 22:26:29.215761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:29.215722 2569 patch_prober.go:28] interesting pod/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 22:26:29.215954 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:29.215781 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 22:26:30.215886 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:30.215844 2569 patch_prober.go:28] interesting pod/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 22:26:30.216315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:30.215908 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 22:26:31.220172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:31.220143 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:31.220663 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:31.220509 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:31.221111 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:31.221093 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:26:31.493827 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:31.493719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:26:51.582772 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:26:51.582735 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:27:01.843636 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:01.843604 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:27:01.847906 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:01.847887 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:27:01.946975 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:01.946942 2569 scope.go:117] "RemoveContainer" containerID="1a786524a028a33f053b0c09d04f57a29360c2043ff12f080d4ee11a1c80a2b5" Jul 23 22:27:27.886824 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:27.886784 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274"] Jul 23 22:27:27.890452 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:27.890435 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:27.893243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:27.893223 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 22:27:27.893376 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:27.893227 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xtsk\"" Jul 23 22:27:27.898124 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:27.897716 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274"] Jul 23 22:27:28.035780 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:28.035747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071748-rs274\" (UID: \"91c8d6a2-54be-4203-894a-a6aa4973926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:28.137089 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:28.136985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071748-rs274\" (UID: \"91c8d6a2-54be-4203-894a-a6aa4973926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:28.137681 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:28.137662 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071748-rs274\" (UID: \"91c8d6a2-54be-4203-894a-a6aa4973926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:28.202695 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:28.202654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:28.337095 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:28.337069 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274"] Jul 23 22:27:28.339524 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:27:28.339475 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c8d6a2_54be_4203_894a_a6aa4973926a.slice/crio-bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528 WatchSource:0}: Error finding container bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528: Status 404 returned error can't find the container with id bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528 Jul 23 22:27:29.120262 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:29.120227 2569 generic.go:356] "Generic (PLEG): container finished" podID="91c8d6a2-54be-4203-894a-a6aa4973926a" containerID="573f1f47093a5ed78fda35df11155597259174b4fe3f736d3d656fbc5d881fe2" exitCode=0 Jul 23 22:27:29.120262 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:29.120267 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" event={"ID":"91c8d6a2-54be-4203-894a-a6aa4973926a","Type":"ContainerDied","Data":"573f1f47093a5ed78fda35df11155597259174b4fe3f736d3d656fbc5d881fe2"} Jul 23 22:27:29.120711 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:29.120319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" event={"ID":"91c8d6a2-54be-4203-894a-a6aa4973926a","Type":"ContainerStarted","Data":"bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528"} Jul 23 22:27:30.253869 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:30.253847 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:30.358531 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:30.358464 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca\") pod \"91c8d6a2-54be-4203-894a-a6aa4973926a\" (UID: \"91c8d6a2-54be-4203-894a-a6aa4973926a\") " Jul 23 22:27:30.358866 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:30.358841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91c8d6a2-54be-4203-894a-a6aa4973926a" (UID: "91c8d6a2-54be-4203-894a-a6aa4973926a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:27:30.459504 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:30.459453 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c8d6a2-54be-4203-894a-a6aa4973926a-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:31.131198 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:31.131152 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" event={"ID":"91c8d6a2-54be-4203-894a-a6aa4973926a","Type":"ContainerDied","Data":"bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528"} Jul 23 22:27:31.131198 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:31.131189 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274" Jul 23 22:27:31.131421 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:31.131232 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb388e82f3809948ceb914044495fabd8a44a1f3f72a5f85ae3133339f705528" Jul 23 22:27:32.098277 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:32.098243 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:27:42.989951 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:42.989913 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274"] Jul 23 22:27:42.992448 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:42.992423 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071748-rs274"] Jul 23 22:27:43.776905 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:43.776871 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c8d6a2-54be-4203-894a-a6aa4973926a" path="/var/lib/kubelet/pods/91c8d6a2-54be-4203-894a-a6aa4973926a/volumes" Jul 23 22:27:47.489666 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:47.489629 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:27:49.984955 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:49.984914 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:27:49.985343 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:49.985121 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" containerID="cri-o://7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70" gracePeriod=30 Jul 23 22:27:54.544164 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:54.544083 2569 patch_prober.go:28] interesting pod/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 22:27:54.544164 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:54.544151 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 22:27:55.238954 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.238926 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:27:55.250443 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.250362 2569 generic.go:356] "Generic (PLEG): container finished" podID="87484ca5-0015-4450-9115-88eeda518f87" containerID="7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70" exitCode=0 Jul 23 22:27:55.250443 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.250415 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" Jul 23 22:27:55.250443 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.250435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerDied","Data":"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70"} Jul 23 22:27:55.250678 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.250496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p" event={"ID":"87484ca5-0015-4450-9115-88eeda518f87","Type":"ContainerDied","Data":"1fd96abd28af7351b600e712a026bc4bdd5046a791626c212de8c69fb23ad87d"} Jul 23 22:27:55.250678 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.250522 2569 scope.go:117] "RemoveContainer" containerID="7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70" Jul 23 22:27:55.259027 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.259007 2569 scope.go:117] "RemoveContainer" containerID="e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058" Jul 23 22:27:55.268606 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.268582 2569 scope.go:117] "RemoveContainer" containerID="7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70" Jul 23 22:27:55.268859 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:27:55.268839 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70\": container with ID starting with 7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70 not found: ID does not exist" containerID="7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70" Jul 23 22:27:55.268930 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.268868 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70"} err="failed to get container status \"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70\": rpc error: code = NotFound desc = could not find container \"7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70\": container with ID starting with 7430a9b52d1dd43a4fb3779c108aa3a2af86597edf71401ea64b48515b177d70 not found: ID does not exist" Jul 23 22:27:55.268930 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.268886 2569 scope.go:117] "RemoveContainer" containerID="e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058" Jul 23 22:27:55.269148 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:27:55.269133 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058\": container with ID starting with e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058 not found: ID does not exist" containerID="e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058" Jul 23 22:27:55.269193 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.269151 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058"} err="failed to get container status \"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058\": rpc error: code = NotFound desc = could not find container \"e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058\": container with ID starting with e976a0fd86d6c867fa7909256f3c03745a8d6da4c686100488d4da61c587f058 not found: ID does not exist" Jul 23 22:27:55.307633 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.307813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.307813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307694 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.307813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307743 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.307813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.307813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307793 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.308071 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307834 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.308071 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307863 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.308071 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.307912 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdktz\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz\") pod \"87484ca5-0015-4450-9115-88eeda518f87\" (UID: \"87484ca5-0015-4450-9115-88eeda518f87\") " Jul 23 22:27:55.308071 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308035 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:27:55.308276 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308168 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:27:55.308332 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308281 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:27:55.308385 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308363 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:27:55.308450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308432 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.308522 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308450 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.308522 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308465 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87484ca5-0015-4450-9115-88eeda518f87-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.308941 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.308915 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data" (OuterVolumeSpecName: "istio-data") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:27:55.309926 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.309896 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token" (OuterVolumeSpecName: "istio-token") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:27:55.310027 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.309956 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:27:55.310377 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.310351 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:27:55.310454 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.310356 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz" (OuterVolumeSpecName: "kube-api-access-xdktz") pod "87484ca5-0015-4450-9115-88eeda518f87" (UID: "87484ca5-0015-4450-9115-88eeda518f87"). InnerVolumeSpecName "kube-api-access-xdktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:27:55.409671 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409629 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.409671 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409665 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdktz\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-kube-api-access-xdktz\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.409671 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409680 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87484ca5-0015-4450-9115-88eeda518f87-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.409915 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409691 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.409915 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409701 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87484ca5-0015-4450-9115-88eeda518f87-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.409915 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.409713 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87484ca5-0015-4450-9115-88eeda518f87-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:27:55.574094 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.574059 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:27:55.579339 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.579314 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-071748-openshift-default-5d7c9b5999-rrz4p"] Jul 23 22:27:55.776853 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:27:55.776820 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87484ca5-0015-4450-9115-88eeda518f87" path="/var/lib/kubelet/pods/87484ca5-0015-4450-9115-88eeda518f87/volumes" Jul 23 22:28:04.190336 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:04.190299 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:28:12.055991 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.055954 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:28:12.056511 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.056220 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" containerID="cri-o://d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595" gracePeriod=30 Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.989614 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990173 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c8d6a2-54be-4203-894a-a6aa4973926a" containerName="revoke-keys" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990194 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c8d6a2-54be-4203-894a-a6aa4973926a" containerName="revoke-keys" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990220 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990228 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990240 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990249 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990328 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:12.991404 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.990346 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c8d6a2-54be-4203-894a-a6aa4973926a" containerName="revoke-keys" Jul 23 22:28:12.994551 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.994526 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:12.997366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:12.997340 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-83cd2f-openshift-default-dockercfg-cwttx\"" Jul 23 22:28:13.008109 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.008083 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:28:13.080747 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.080747 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.080747 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rbv\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.080938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.081025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.081361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.081066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182143 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182143 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rbv\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182396 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182818 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182818 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182679 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182818 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182698 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182923 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182857 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.182993 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.182969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.184709 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.184686 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.184900 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.184886 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.190805 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.190781 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.191158 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.191141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rbv\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv\") pod \"e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.304624 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.304532 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:13.446025 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.445994 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:28:13.448035 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:28:13.448007 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d550f2_b340_4b5e_babc_4fcf442c21b1.slice/crio-6b9213239b10e250ac3932fd0c902727e544088444f6721cfd395ed8846d6c13 WatchSource:0}: Error finding container 6b9213239b10e250ac3932fd0c902727e544088444f6721cfd395ed8846d6c13: Status 404 returned error can't find the container with id 6b9213239b10e250ac3932fd0c902727e544088444f6721cfd395ed8846d6c13 Jul 23 22:28:13.450560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.450525 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:13.450634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.450607 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:13.450679 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:13.450635 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:14.345620 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:14.345587 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerStarted","Data":"88cfaa1635a10dbc17acf1a1b40cf7a3a8dfa2c8ff4698b12dab4638cdeb6443"} Jul 23 22:28:14.346011 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:14.345634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerStarted","Data":"6b9213239b10e250ac3932fd0c902727e544088444f6721cfd395ed8846d6c13"} Jul 23 22:28:14.366796 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:14.366742 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podStartSLOduration=2.366726764 podStartE2EDuration="2.366726764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:14.365797837 +0000 UTC m=+2174.391131408" watchObservedRunningTime="2026-07-23 22:28:14.366726764 +0000 UTC m=+2174.392060299" Jul 23 22:28:15.305395 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:15.305357 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:15.310150 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:15.310127 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:15.350394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:15.350364 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:15.351354 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:15.351337 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:16.223285 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:16.223245 2569 patch_prober.go:28] interesting pod/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 22:28:16.223458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:16.223305 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 22:28:17.324408 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.324343 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:28:17.362867 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.362829 2569 generic.go:356] "Generic (PLEG): container finished" podID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerID="d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595" exitCode=0 Jul 23 22:28:17.363063 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.362887 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" Jul 23 22:28:17.363063 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.362933 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerDied","Data":"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595"} Jul 23 22:28:17.363063 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.362996 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm" event={"ID":"f14fa5d4-e3db-45a7-b039-faa01b45432d","Type":"ContainerDied","Data":"4a6ff57f0005804195edcb31397479563f9e06c1f1c9a6ffbee2813a0ce4150c"} Jul 23 22:28:17.363063 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.363027 2569 scope.go:117] "RemoveContainer" containerID="d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595" Jul 23 22:28:17.372687 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.372665 2569 scope.go:117] "RemoveContainer" containerID="270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c" Jul 23 22:28:17.381561 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.381539 2569 scope.go:117] "RemoveContainer" containerID="d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595" Jul 23 22:28:17.381837 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:28:17.381819 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595\": container with ID starting with d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595 not found: ID does not exist" containerID="d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595" Jul 23 22:28:17.381896 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.381846 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595"} err="failed to get container status \"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595\": rpc error: code = NotFound desc = could not find container \"d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595\": container with ID starting with d4000849a7d0b65d90c5360f8229c063421fdf029a4302f775c5b3512c1cf595 not found: ID does not exist" Jul 23 22:28:17.381896 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.381866 2569 scope.go:117] "RemoveContainer" containerID="270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c" Jul 23 22:28:17.382105 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:28:17.382086 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c\": container with ID starting with 270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c not found: ID does not exist" containerID="270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c" Jul 23 22:28:17.382161 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.382111 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c"} err="failed to get container status \"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c\": rpc error: code = NotFound desc = could not find container \"270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c\": container with ID starting with 270e64efe7ce048fe04ab21322bc9fb6986dbab40b0939bb9de829039f16132c not found: ID does not exist" Jul 23 22:28:17.425719 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425687 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.425852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425744 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.425852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425766 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.425852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425801 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.425852 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425829 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425871 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425906 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425933 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nctn\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.425956 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert\") pod \"f14fa5d4-e3db-45a7-b039-faa01b45432d\" (UID: \"f14fa5d4-e3db-45a7-b039-faa01b45432d\") " Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426015 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:28:17.426061 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426044 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:28:17.426579 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426349 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.426579 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426369 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.426579 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426431 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:28:17.426940 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426916 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:28:17.427009 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.426991 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:28:17.428067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.428043 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:28:17.428501 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.428467 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn" (OuterVolumeSpecName: "kube-api-access-2nctn") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "kube-api-access-2nctn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:28:17.428602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.428541 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:28:17.428855 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.428836 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f14fa5d4-e3db-45a7-b039-faa01b45432d" (UID: "f14fa5d4-e3db-45a7-b039-faa01b45432d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:28:17.527366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527331 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527361 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527370 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527656 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527378 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nctn\" (UniqueName: \"kubernetes.io/projected/f14fa5d4-e3db-45a7-b039-faa01b45432d-kube-api-access-2nctn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527656 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527387 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14fa5d4-e3db-45a7-b039-faa01b45432d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527656 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527395 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.527656 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.527402 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14fa5d4-e3db-45a7-b039-faa01b45432d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:28:17.687687 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.687658 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:28:17.691252 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.691210 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b12a8c-openshift-default-58ccbb9645-qt2hm"] Jul 23 22:28:17.776721 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:17.776680 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" path="/var/lib/kubelet/pods/f14fa5d4-e3db-45a7-b039-faa01b45432d/volumes" Jul 23 22:28:21.666771 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.666742 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:28:21.667153 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.667102 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:28:21.667153 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.667112 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:28:21.667272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.667253 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="87484ca5-0015-4450-9115-88eeda518f87" containerName="istio-proxy" Jul 23 22:28:21.667272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.667270 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:28:21.671883 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.671861 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.674579 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.674554 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9684d1-openshift-default-dockercfg-vg425\"" Jul 23 22:28:21.683251 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.683207 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:28:21.769333 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769301 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769333 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7w9n\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769590 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.769794 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.769606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870657 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870616 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7w9n\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.870863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.870838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871438 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871438 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871346 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871574 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871464 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.871637 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.871612 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.873422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.873396 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.873559 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.873541 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.878279 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.878254 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.878372 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.878325 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7w9n\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n\") pod \"e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:21.983502 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:21.983389 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:22.116173 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.116144 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:28:22.117813 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:28:22.117781 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60f1d3d_b030_460b_a1ed_ad89dd22a62b.slice/crio-7f938fb00ced488184436c52d726795f3934c9d24155af8531880e144f78bd71 WatchSource:0}: Error finding container 7f938fb00ced488184436c52d726795f3934c9d24155af8531880e144f78bd71: Status 404 returned error can't find the container with id 7f938fb00ced488184436c52d726795f3934c9d24155af8531880e144f78bd71 Jul 23 22:28:22.120133 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.120085 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:22.120241 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.120188 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:22.120294 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.120238 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:22.396575 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.396535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerStarted","Data":"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb"} Jul 23 22:28:22.396763 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.396625 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerStarted","Data":"7f938fb00ced488184436c52d726795f3934c9d24155af8531880e144f78bd71"} Jul 23 22:28:22.421861 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.421808 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podStartSLOduration=1.421793215 podStartE2EDuration="1.421793215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:22.420746319 +0000 UTC m=+2182.446079855" watchObservedRunningTime="2026-07-23 22:28:22.421793215 +0000 UTC m=+2182.447126728" Jul 23 22:28:22.984241 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:22.984202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:23.988620 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:23.988590 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:24.405862 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:24.405833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:24.406708 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:24.406685 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:24.592239 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:24.592205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:28:31.392726 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:31.392677 2569 patch_prober.go:28] interesting pod/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 22:28:31.393142 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:31.392763 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 22:28:33.455986 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.455952 2569 generic.go:356] "Generic (PLEG): container finished" podID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerID="88cfaa1635a10dbc17acf1a1b40cf7a3a8dfa2c8ff4698b12dab4638cdeb6443" exitCode=0 Jul 23 22:28:33.456458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.456026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerDied","Data":"88cfaa1635a10dbc17acf1a1b40cf7a3a8dfa2c8ff4698b12dab4638cdeb6443"} Jul 23 22:28:33.456458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.456449 2569 scope.go:117] "RemoveContainer" containerID="88cfaa1635a10dbc17acf1a1b40cf7a3a8dfa2c8ff4698b12dab4638cdeb6443" Jul 23 22:28:33.457469 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.457439 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:33.457560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.457524 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:33.457560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:33.457555 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:34.463338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:34.463299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerStarted","Data":"0a0e54aaaaf79fdda39b78ad2ead1fe59e573b3d8808d5dd8246dff29d2eb9a8"} Jul 23 22:28:35.305921 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:35.305874 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:35.306145 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:35.306018 2569 patch_prober.go:28] interesting pod/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 22:28:35.306145 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:35.306062 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 22:28:36.305687 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:36.305637 2569 patch_prober.go:28] interesting pod/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 22:28:36.306191 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:36.305710 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 22:28:37.305905 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:37.305864 2569 patch_prober.go:28] interesting pod/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 22:28:37.306518 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:37.306461 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 22:28:38.305801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:38.305710 2569 patch_prober.go:28] interesting pod/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 22:28:38.306065 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:38.306036 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 22:28:39.308495 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:39.308442 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:39.308889 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:39.308810 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:39.309443 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:39.309423 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:28:39.393568 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:39.393533 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:28:40.447687 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:40.447646 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 22:28:40.449083 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:40.447739 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 22:28:41.502896 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.502865 2569 generic.go:356] "Generic (PLEG): container finished" podID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerID="82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb" exitCode=0 Jul 23 22:28:41.503299 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.502939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerDied","Data":"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb"} Jul 23 22:28:41.503366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.503351 2569 scope.go:117] "RemoveContainer" containerID="82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb" Jul 23 22:28:41.504432 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.504406 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:41.504498 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.504473 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:41.504545 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:41.504520 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:28:42.514034 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:42.513993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerStarted","Data":"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031"} Jul 23 22:28:42.983872 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:42.983829 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:42.984067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:42.984042 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 22:28:42.984115 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:42.984099 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 22:28:43.983990 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:43.983949 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 22:28:43.984391 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:43.984009 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 22:28:44.984194 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:44.984160 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 22:28:44.984622 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:44.984213 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 22:28:45.983998 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:45.983954 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 22:28:45.984189 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:45.984028 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 22:28:46.987701 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:46.987669 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:46.988080 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:46.987909 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:46.988670 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:46.988654 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:28:55.892064 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:28:55.892027 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:29:01.978633 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:01.978599 2569 scope.go:117] "RemoveContainer" containerID="7e0841c3fb0dfc7b454674d135c6719deb318d8c55215a59afa8990fad31ceb5" Jul 23 22:29:03.517262 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:03.517229 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:29:03.517684 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:03.517428 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" containerID="cri-o://0a0e54aaaaf79fdda39b78ad2ead1fe59e573b3d8808d5dd8246dff29d2eb9a8" gracePeriod=30 Jul 23 22:29:08.647682 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.647650 2569 generic.go:356] "Generic (PLEG): container finished" podID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerID="0a0e54aaaaf79fdda39b78ad2ead1fe59e573b3d8808d5dd8246dff29d2eb9a8" exitCode=0 Jul 23 22:29:08.648154 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.647694 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerDied","Data":"0a0e54aaaaf79fdda39b78ad2ead1fe59e573b3d8808d5dd8246dff29d2eb9a8"} Jul 23 22:29:08.648154 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.647751 2569 scope.go:117] "RemoveContainer" containerID="88cfaa1635a10dbc17acf1a1b40cf7a3a8dfa2c8ff4698b12dab4638cdeb6443" Jul 23 22:29:08.775985 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.775954 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:29:08.973194 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973147 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973212 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973230 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973246 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973290 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rbv\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973359 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973394 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973384 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973411 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket\") pod \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\" (UID: \"f9d550f2-b340-4b5e-babc-4fcf442c21b1\") " Jul 23 22:29:08.973801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973451 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:08.973801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973578 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:29:08.973801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973740 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:08.973801 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973792 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:08.974086 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.973820 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:08.974141 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.974107 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:08.974366 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.974340 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:08.975638 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.975612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:29:08.975746 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.975634 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:08.975903 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.975877 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:29:08.975967 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:08.975881 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv" (OuterVolumeSpecName: "kube-api-access-g8rbv") pod "f9d550f2-b340-4b5e-babc-4fcf442c21b1" (UID: "f9d550f2-b340-4b5e-babc-4fcf442c21b1"). InnerVolumeSpecName "kube-api-access-g8rbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:29:09.074849 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074794 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.074849 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074843 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.074849 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074858 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.075108 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074872 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rbv\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-kube-api-access-g8rbv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.075108 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074884 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.075108 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074894 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d550f2-b340-4b5e-babc-4fcf442c21b1-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.075108 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.074905 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d550f2-b340-4b5e-babc-4fcf442c21b1-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:09.654098 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.654070 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" Jul 23 22:29:09.654597 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.654069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh" event={"ID":"f9d550f2-b340-4b5e-babc-4fcf442c21b1","Type":"ContainerDied","Data":"6b9213239b10e250ac3932fd0c902727e544088444f6721cfd395ed8846d6c13"} Jul 23 22:29:09.654597 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.654197 2569 scope.go:117] "RemoveContainer" containerID="0a0e54aaaaf79fdda39b78ad2ead1fe59e573b3d8808d5dd8246dff29d2eb9a8" Jul 23 22:29:09.677657 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.677624 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:29:09.682155 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.682091 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-83cd2f-openshift-default-5bb9cf8f55-2rjjh"] Jul 23 22:29:09.776755 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:09.776723 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" path="/var/lib/kubelet/pods/f9d550f2-b340-4b5e-babc-4fcf442c21b1/volumes" Jul 23 22:29:23.184293 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:23.184207 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:29:30.792025 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:30.791991 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:29:30.792441 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:30.792215 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" containerID="cri-o://9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031" gracePeriod=30 Jul 23 22:29:31.736107 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736076 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:29:31.736464 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736452 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:29:31.736541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736467 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:29:31.736541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736497 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.736541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736503 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.736541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736517 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.736541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736523 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.736717 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736591 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14fa5d4-e3db-45a7-b039-faa01b45432d" containerName="istio-proxy" Jul 23 22:29:31.736717 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736603 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.736717 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.736609 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d550f2-b340-4b5e-babc-4fcf442c21b1" containerName="istio-proxy" Jul 23 22:29:31.739600 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.739580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.743099 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.743080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1c273d-openshift-default-dockercfg-cbxj2\"" Jul 23 22:29:31.757088 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.757065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:29:31.876916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.876879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.876916 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.876914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2w2\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.876942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.876961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.877042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.877099 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.877153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.877190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.877407 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.877265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2w2\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.978864 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978859 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.979067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978904 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.979067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.979067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.978969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.979242 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.979207 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.979556 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.979535 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.980832 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.980812 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.981177 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.981154 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.986984 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.986925 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.987315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.987298 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2w2\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2\") pod \"e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:31.990553 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.990506 2569 patch_prober.go:28] interesting pod/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 22:29:31.990648 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:31.990565 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 22:29:32.050499 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.050457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:32.186848 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.186823 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:29:32.189079 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:29:32.189046 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd7e8c3_bcc9_474b_895c_9732162d4da5.slice/crio-eb0967d59f57cb39fec53da6b4fd52537d28de05166468e6b2fccc282add1f01 WatchSource:0}: Error finding container eb0967d59f57cb39fec53da6b4fd52537d28de05166468e6b2fccc282add1f01: Status 404 returned error can't find the container with id eb0967d59f57cb39fec53da6b4fd52537d28de05166468e6b2fccc282add1f01 Jul 23 22:29:32.191229 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.191197 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:32.191331 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.191256 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:32.191331 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.191285 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:32.771937 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.771904 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerStarted","Data":"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8"} Jul 23 22:29:32.771937 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.771944 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerStarted","Data":"eb0967d59f57cb39fec53da6b4fd52537d28de05166468e6b2fccc282add1f01"} Jul 23 22:29:32.794191 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:32.794138 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podStartSLOduration=1.7941225379999999 podStartE2EDuration="1.794122538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:29:32.793108627 +0000 UTC m=+2252.818442186" watchObservedRunningTime="2026-07-23 22:29:32.794122538 +0000 UTC m=+2252.819456073" Jul 23 22:29:33.051458 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:33.051366 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:33.056547 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:33.056523 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:33.776534 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:33.776501 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:33.777341 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:33.777324 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:36.045848 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.045815 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:29:36.134355 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134540 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134370 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134540 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134404 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134540 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134433 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134540 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134464 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134549 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134574 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134601 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7w9n\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134634 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134661 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo\") pod \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\" (UID: \"e60f1d3d-b030-460b-a1ed-ad89dd22a62b\") " Jul 23 22:29:36.134767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134749 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:36.135068 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134874 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:29:36.135068 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.134977 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:36.135165 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.135077 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.135165 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.135094 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.135165 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.135108 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.135165 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.135121 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.135452 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.135421 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data" (OuterVolumeSpecName: "istio-data") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:36.137001 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.136969 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n" (OuterVolumeSpecName: "kube-api-access-j7w9n") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "kube-api-access-j7w9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:29:36.137086 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.136997 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:29:36.137195 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.137176 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:29:36.137235 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.137197 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token" (OuterVolumeSpecName: "istio-token") pod "e60f1d3d-b030-460b-a1ed-ad89dd22a62b" (UID: "e60f1d3d-b030-460b-a1ed-ad89dd22a62b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:29:36.235765 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.235730 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.235765 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.235758 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.235765 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.235767 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.235765 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.235775 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7w9n\" (UniqueName: \"kubernetes.io/projected/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-kube-api-access-j7w9n\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.236036 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.235784 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f1d3d-b030-460b-a1ed-ad89dd22a62b-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:29:36.792951 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.792916 2569 generic.go:356] "Generic (PLEG): container finished" podID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerID="9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031" exitCode=0 Jul 23 22:29:36.792951 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.792952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerDied","Data":"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031"} Jul 23 22:29:36.793189 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.792960 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" Jul 23 22:29:36.793189 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.792986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j" event={"ID":"e60f1d3d-b030-460b-a1ed-ad89dd22a62b","Type":"ContainerDied","Data":"7f938fb00ced488184436c52d726795f3934c9d24155af8531880e144f78bd71"} Jul 23 22:29:36.793189 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.793009 2569 scope.go:117] "RemoveContainer" containerID="9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031" Jul 23 22:29:36.802314 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.802298 2569 scope.go:117] "RemoveContainer" containerID="82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb" Jul 23 22:29:36.812517 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.812493 2569 scope.go:117] "RemoveContainer" containerID="9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031" Jul 23 22:29:36.812827 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:29:36.812805 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031\": container with ID starting with 9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031 not found: ID does not exist" containerID="9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031" Jul 23 22:29:36.812919 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.812839 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031"} err="failed to get container status \"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031\": rpc error: code = NotFound desc = could not find container \"9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031\": container with ID starting with 9128d320b5e1cc4c2dd2e4892a0ba013446d9c6b9b5ac9beea9d11b296f75031 not found: ID does not exist" Jul 23 22:29:36.812919 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.812863 2569 scope.go:117] "RemoveContainer" containerID="82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb" Jul 23 22:29:36.813173 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:29:36.813150 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb\": container with ID starting with 82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb not found: ID does not exist" containerID="82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb" Jul 23 22:29:36.813240 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.813180 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb"} err="failed to get container status \"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb\": rpc error: code = NotFound desc = could not find container \"82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb\": container with ID starting with 82fda53f7f389d9d4006a7f760536316a9ddf395ebfed265c9d5d42017b070cb not found: ID does not exist" Jul 23 22:29:36.816806 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.816778 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:29:36.823560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:36.823538 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9684d1-openshift-default-7dc9c6d585-gq82j"] Jul 23 22:29:37.776978 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:37.776947 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" path="/var/lib/kubelet/pods/e60f1d3d-b030-460b-a1ed-ad89dd22a62b/volumes" Jul 23 22:29:37.889048 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:37.889012 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:29:39.938813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.938766 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:29:39.939442 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.939345 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:29:39.939442 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.939363 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:29:39.939618 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.939473 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:29:39.943323 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.943298 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:39.945822 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.945798 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6bf8de-openshift-default-dockercfg-ljxnb\"" Jul 23 22:29:39.956138 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:39.956114 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:29:40.076023 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.075986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076023 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076373 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076373 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.076373 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.076316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fsh\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177360 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177602 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177870 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fsh\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177870 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.177870 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.178046 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.178046 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.177980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.178164 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.178061 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.178164 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.178148 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.178266 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.178194 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.179797 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.179772 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.180409 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.180388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.185897 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.185874 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.185999 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.185931 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fsh\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh\") pod \"e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.254949 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.254848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:40.432243 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.432214 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:29:40.434545 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:29:40.434518 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5801076b_9446_4a26_9e17_24a6acc73db3.slice/crio-dabc80f41b3a875711227530d059bb6481116112c56e0c9e5d6365ed4f655e26 WatchSource:0}: Error finding container dabc80f41b3a875711227530d059bb6481116112c56e0c9e5d6365ed4f655e26: Status 404 returned error can't find the container with id dabc80f41b3a875711227530d059bb6481116112c56e0c9e5d6365ed4f655e26 Jul 23 22:29:40.436740 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.436705 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:40.436821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.436773 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:40.436821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.436805 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:40.821950 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.821864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerStarted","Data":"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050"} Jul 23 22:29:40.821950 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.821913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerStarted","Data":"dabc80f41b3a875711227530d059bb6481116112c56e0c9e5d6365ed4f655e26"} Jul 23 22:29:40.848592 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:40.848528 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podStartSLOduration=1.848507873 podStartE2EDuration="1.848507873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:29:40.846854876 +0000 UTC m=+2260.872188410" watchObservedRunningTime="2026-07-23 22:29:40.848507873 +0000 UTC m=+2260.873841411" Jul 23 22:29:41.255508 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:41.255444 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:41.260585 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:41.260561 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:41.826829 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:41.826799 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:41.827829 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:41.827810 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:29:45.847696 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.847663 2569 generic.go:356] "Generic (PLEG): container finished" podID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerID="4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8" exitCode=0 Jul 23 22:29:45.848122 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.847738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerDied","Data":"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8"} Jul 23 22:29:45.848168 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.848121 2569 scope.go:117] "RemoveContainer" containerID="4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8" Jul 23 22:29:45.849187 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.849151 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:45.849295 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.849221 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:45.849295 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:45.849266 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:29:46.854717 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:46.854681 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerStarted","Data":"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911"} Jul 23 22:29:47.051258 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:47.051222 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:47.052123 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:47.052098 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 22:29:47.052187 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:47.052161 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 22:29:48.050997 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:48.050961 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 22:29:48.051400 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:48.051024 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 22:29:48.777188 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:48.777154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:49.051621 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:49.051535 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 22:29:49.051621 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:49.051613 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 22:29:50.051840 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:50.051800 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 22:29:50.052338 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:50.051878 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 22:29:51.055242 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:51.055213 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:51.056232 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:51.056214 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:29:52.689102 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:29:52.689069 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:30:00.931560 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.931525 2569 generic.go:356] "Generic (PLEG): container finished" podID="5801076b-9446-4a26-9e17-24a6acc73db3" containerID="d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050" exitCode=0 Jul 23 22:30:00.932102 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.931608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerDied","Data":"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050"} Jul 23 22:30:00.932179 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.932128 2569 scope.go:117] "RemoveContainer" containerID="d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050" Jul 23 22:30:00.933147 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.933119 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:30:00.933247 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.933188 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:30:00.933247 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:00.933219 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 22:30:01.938180 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:01.938145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerStarted","Data":"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f"} Jul 23 22:30:02.004083 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:02.004042 2569 scope.go:117] "RemoveContainer" containerID="61c51f2bf0b8b7417728dde834fac51ef9a0509d83aaa196d8b60833e1c72973" Jul 23 22:30:02.255142 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:02.255052 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:30:02.255342 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:02.255316 2569 patch_prober.go:28] interesting pod/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 22:30:02.255386 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:02.255367 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 22:30:03.256214 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:03.256177 2569 patch_prober.go:28] interesting pod/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 22:30:03.256658 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:03.256238 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 22:30:04.255863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:04.255823 2569 patch_prober.go:28] interesting pod/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 22:30:04.256052 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:04.255880 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 22:30:05.256249 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:05.256207 2569 patch_prober.go:28] interesting pod/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 22:30:05.256682 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:05.256279 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 22:30:06.259177 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:06.259147 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:30:06.259576 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:06.259432 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:30:06.260032 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:06.260016 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:30:07.892894 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:07.892861 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:30:34.487933 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:34.487895 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:30:51.683256 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:30:51.683220 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:31:09.537608 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:31:09.537570 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:31:26.389402 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:31:26.389362 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:31:43.890103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:31:43.890065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:32:01.889410 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:01.889379 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:32:01.893589 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:01.893568 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:32:01.993908 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:01.993875 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:32:19.179000 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:19.178960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:32:35.388153 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:35.388064 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:32:49.189791 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:32:49.189752 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:33:27.888813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:33:27.888777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:33:46.985194 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:33:46.985153 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:34:00.484818 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:34:00.484734 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:34:02.064620 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:34:02.064588 2569 scope.go:117] "RemoveContainer" containerID="573f1f47093a5ed78fda35df11155597259174b4fe3f736d3d656fbc5d881fe2" Jul 23 22:34:20.193853 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:34:20.193819 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:34:53.193770 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:34:53.193735 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:35:13.117428 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:35:13.117391 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:35:29.087807 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:35:29.087724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:35:49.282055 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:35:49.282016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:36:08.695077 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:08.695041 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:36:16.970600 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:16.970564 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:36:16.971115 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:16.970781 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" containerID="cri-o://934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911" gracePeriod=30 Jul 23 22:36:21.058450 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:21.058404 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 22:36:21.058881 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:21.058463 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 22:36:22.230767 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.230694 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:36:22.299030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.298992 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299040 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299102 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt2w2\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299197 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299239 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299268 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299315 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299296 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299615 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299349 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299615 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299378 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs\") pod \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\" (UID: \"dbd7e8c3-bcc9-474b-895c-9732162d4da5\") " Jul 23 22:36:22.299615 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299553 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:36:22.299888 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299834 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:22.299888 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299851 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.299888 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299848 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:22.299888 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.299639 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:22.300352 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.300322 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:22.301541 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.301510 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2" (OuterVolumeSpecName: "kube-api-access-dt2w2") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "kube-api-access-dt2w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:36:22.301651 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.301614 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:36:22.301800 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.301776 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:36:22.301800 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.301785 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbd7e8c3-bcc9-474b-895c-9732162d4da5" (UID: "dbd7e8c3-bcc9-474b-895c-9732162d4da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:22.401434 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401389 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2w2\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-kube-api-access-dt2w2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401434 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401426 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401434 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401436 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401434 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401445 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401755 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401453 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401755 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401461 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401755 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401470 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd7e8c3-bcc9-474b-895c-9732162d4da5-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.401755 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.401508 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd7e8c3-bcc9-474b-895c-9732162d4da5-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:22.775445 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.775410 2569 generic.go:356] "Generic (PLEG): container finished" podID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerID="934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911" exitCode=0 Jul 23 22:36:22.775634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.775457 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" Jul 23 22:36:22.775634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.775524 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerDied","Data":"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911"} Jul 23 22:36:22.775634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.775575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc" event={"ID":"dbd7e8c3-bcc9-474b-895c-9732162d4da5","Type":"ContainerDied","Data":"eb0967d59f57cb39fec53da6b4fd52537d28de05166468e6b2fccc282add1f01"} Jul 23 22:36:22.775634 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.775605 2569 scope.go:117] "RemoveContainer" containerID="934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911" Jul 23 22:36:22.784953 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.784934 2569 scope.go:117] "RemoveContainer" containerID="4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8" Jul 23 22:36:22.793021 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.792999 2569 scope.go:117] "RemoveContainer" containerID="934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911" Jul 23 22:36:22.793269 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:36:22.793248 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911\": container with ID starting with 934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911 not found: ID does not exist" containerID="934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911" Jul 23 22:36:22.793305 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.793281 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911"} err="failed to get container status \"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911\": rpc error: code = NotFound desc = could not find container \"934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911\": container with ID starting with 934f1fd619ca78caa6059a300229485379ec9282f2a0d2b195dbbb4172f94911 not found: ID does not exist" Jul 23 22:36:22.793305 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.793299 2569 scope.go:117] "RemoveContainer" containerID="4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8" Jul 23 22:36:22.793524 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:36:22.793507 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8\": container with ID starting with 4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8 not found: ID does not exist" containerID="4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8" Jul 23 22:36:22.793562 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.793534 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8"} err="failed to get container status \"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8\": rpc error: code = NotFound desc = could not find container \"4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8\": container with ID starting with 4eb30016f652513790dd2d778effc8e229067b86c5f685a3c618bcc39b3a0dd8 not found: ID does not exist" Jul 23 22:36:22.797462 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.797441 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:36:22.800298 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:22.800275 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1c273d-openshift-default-6bfdf5f87-b8mkc"] Jul 23 22:36:23.777369 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:23.777335 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" path="/var/lib/kubelet/pods/dbd7e8c3-bcc9-474b-895c-9732162d4da5/volumes" Jul 23 22:36:29.587410 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:29.587370 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pxclx"] Jul 23 22:36:39.110875 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:39.110836 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:36:39.111284 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:39.111058 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" containerID="cri-o://31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f" gracePeriod=30 Jul 23 22:36:44.371726 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.371655 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:36:44.425754 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425717 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.425932 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425770 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.425932 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425797 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fsh\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.425932 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425900 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425942 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425971 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.425993 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426103 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426075 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket\") pod \"5801076b-9446-4a26-9e17-24a6acc73db3\" (UID: \"5801076b-9446-4a26-9e17-24a6acc73db3\") " Jul 23 22:36:44.426361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426241 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:44.426361 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426335 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 22:36:44.426506 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426462 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5801076b-9446-4a26-9e17-24a6acc73db3-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.426600 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426500 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:44.426600 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426514 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.426717 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426696 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:44.426844 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.426804 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data" (OuterVolumeSpecName: "istio-data") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:44.428158 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.428132 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 22:36:44.428349 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.428322 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh" (OuterVolumeSpecName: "kube-api-access-47fsh") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "kube-api-access-47fsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:36:44.428444 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.428369 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 22:36:44.428846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.428819 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5801076b-9446-4a26-9e17-24a6acc73db3" (UID: "5801076b-9446-4a26-9e17-24a6acc73db3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 22:36:44.527568 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527528 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527568 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527563 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527568 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527572 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527586 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527596 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5801076b-9446-4a26-9e17-24a6acc73db3-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527604 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5801076b-9446-4a26-9e17-24a6acc73db3-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.527821 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.527612 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fsh\" (UniqueName: \"kubernetes.io/projected/5801076b-9446-4a26-9e17-24a6acc73db3-kube-api-access-47fsh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Jul 23 22:36:44.883389 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.883354 2569 generic.go:356] "Generic (PLEG): container finished" podID="5801076b-9446-4a26-9e17-24a6acc73db3" containerID="31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f" exitCode=0 Jul 23 22:36:44.883614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.883397 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" Jul 23 22:36:44.883614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.883398 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerDied","Data":"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f"} Jul 23 22:36:44.883614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.883541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb" event={"ID":"5801076b-9446-4a26-9e17-24a6acc73db3","Type":"ContainerDied","Data":"dabc80f41b3a875711227530d059bb6481116112c56e0c9e5d6365ed4f655e26"} Jul 23 22:36:44.883614 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.883579 2569 scope.go:117] "RemoveContainer" containerID="31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f" Jul 23 22:36:44.893355 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.893336 2569 scope.go:117] "RemoveContainer" containerID="d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050" Jul 23 22:36:44.904718 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.904448 2569 scope.go:117] "RemoveContainer" containerID="31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f" Jul 23 22:36:44.905312 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:36:44.905283 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f\": container with ID starting with 31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f not found: ID does not exist" containerID="31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f" Jul 23 22:36:44.905414 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.905324 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f"} err="failed to get container status \"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f\": rpc error: code = NotFound desc = could not find container \"31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f\": container with ID starting with 31f61ea896e36a251a1f2a21c6b17dbf3fd6462aa56b6752178f4aa60e10168f not found: ID does not exist" Jul 23 22:36:44.905414 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.905349 2569 scope.go:117] "RemoveContainer" containerID="d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050" Jul 23 22:36:44.905695 ip-10-0-142-110 kubenswrapper[2569]: E0723 22:36:44.905677 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050\": container with ID starting with d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050 not found: ID does not exist" containerID="d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050" Jul 23 22:36:44.905768 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.905705 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050"} err="failed to get container status \"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050\": rpc error: code = NotFound desc = could not find container \"d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050\": container with ID starting with d2f826e59fbba912bb0acb36f2efb2cc250b713934c52e8ff807114bab4f4050 not found: ID does not exist" Jul 23 22:36:44.907535 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.907513 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:36:44.910721 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:44.910699 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf8de-openshift-default-7b9554564c-slqdb"] Jul 23 22:36:45.777272 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:45.777237 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" path="/var/lib/kubelet/pods/5801076b-9446-4a26-9e17-24a6acc73db3/volumes" Jul 23 22:36:46.256863 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:46.256834 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp_35547a13-180d-475d-9a6e-934a0090328a/copy-manifests/0.log" Jul 23 22:36:46.264468 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:46.264415 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp_35547a13-180d-475d-9a6e-934a0090328a/manager/0.log" Jul 23 22:36:46.896469 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:46.896439 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747415-m4f7k_0cadaf9c-8ee1-40d8-9ffb-24217c852e29/cleanup/0.log" Jul 23 22:36:49.270447 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:49.270417 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pxclx_5b164ffa-ab2a-40fe-bfa8-c0e37722a547/limitador/0.log" Jul 23 22:36:49.760884 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:49.760850 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l4d8d_b0e6693d-0015-41f2-bf37-b09cb5aa177e/istio-proxy/0.log" Jul 23 22:36:49.893761 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:49.893727 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvs82_8ede2f35-344b-43a1-94e3-b86f569bb891/discovery/0.log" Jul 23 22:36:50.257360 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:36:50.257325 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/1.log" Jul 23 22:37:00.893285 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:00.893204 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b5sb5_1ab2eb30-dc87-4854-bd4a-dbbc7a68e1da/global-pull-secret-syncer/0.log" Jul 23 22:37:01.031645 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:01.031611 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p6f5d_6869373b-5e3c-428e-bd23-e8592620ef52/konnectivity-agent/0.log" Jul 23 22:37:01.114500 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:01.114451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-110.ec2.internal_72d12a7c9f3bb00f63b0202d19fa6945/haproxy/0.log" Jul 23 22:37:01.920618 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:01.920587 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:37:01.924770 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:01.924746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf658_5fd867b5-4c02-41fb-9b89-c95297ed5dd3/ovn-acl-logging/0.log" Jul 23 22:37:07.004651 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:07.004621 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pxclx_5b164ffa-ab2a-40fe-bfa8-c0e37722a547/limitador/0.log" Jul 23 22:37:08.683096 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.683067 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/alertmanager/0.log" Jul 23 22:37:08.719813 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.719787 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/config-reloader/0.log" Jul 23 22:37:08.758853 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.758825 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/kube-rbac-proxy-web/0.log" Jul 23 22:37:08.800682 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.800647 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/kube-rbac-proxy/0.log" Jul 23 22:37:08.830658 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.830629 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/kube-rbac-proxy-metric/0.log" Jul 23 22:37:08.873781 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.873755 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/prom-label-proxy/0.log" Jul 23 22:37:08.909845 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:08.909806 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3a624481-7a31-4e43-8e65-434b23781bef/init-config-reloader/0.log" Jul 23 22:37:09.023204 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.023103 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m9tj6_75ab352b-901d-4f9b-9d4e-e413a4a0d0ae/kube-state-metrics/0.log" Jul 23 22:37:09.061884 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.061847 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m9tj6_75ab352b-901d-4f9b-9d4e-e413a4a0d0ae/kube-rbac-proxy-main/0.log" Jul 23 22:37:09.103848 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.103815 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m9tj6_75ab352b-901d-4f9b-9d4e-e413a4a0d0ae/kube-rbac-proxy-self/0.log" Jul 23 22:37:09.145347 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.145315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5cc8ffdcb8-k42xl_94d9f069-4bf8-4d3e-a534-7e48762f50f2/metrics-server/0.log" Jul 23 22:37:09.169931 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.169903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-tnxwm_34bbc941-25c2-4878-8709-249385e4adec/monitoring-plugin/0.log" Jul 23 22:37:09.396515 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.396471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xt4lv_086846b1-3bf5-4845-af98-154e46dd4fac/node-exporter/0.log" Jul 23 22:37:09.417596 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.417567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xt4lv_086846b1-3bf5-4845-af98-154e46dd4fac/kube-rbac-proxy/0.log" Jul 23 22:37:09.450037 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.450013 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xt4lv_086846b1-3bf5-4845-af98-154e46dd4fac/init-textfile/0.log" Jul 23 22:37:09.476904 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.476881 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-stxw6_25aecb24-e466-4359-9884-c887b54474e3/kube-rbac-proxy-main/0.log" Jul 23 22:37:09.500449 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.500419 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-stxw6_25aecb24-e466-4359-9884-c887b54474e3/kube-rbac-proxy-self/0.log" Jul 23 22:37:09.524247 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.524212 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-stxw6_25aecb24-e466-4359-9884-c887b54474e3/openshift-state-metrics/0.log" Jul 23 22:37:09.556862 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.556823 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/prometheus/0.log" Jul 23 22:37:09.576352 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.576322 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/config-reloader/0.log" Jul 23 22:37:09.600932 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.600851 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/thanos-sidecar/0.log" Jul 23 22:37:09.620707 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.620673 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/kube-rbac-proxy-web/0.log" Jul 23 22:37:09.640837 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.640811 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/kube-rbac-proxy/0.log" Jul 23 22:37:09.682030 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.681994 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/kube-rbac-proxy-thanos/0.log" Jul 23 22:37:09.703223 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.703194 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bcca40-0c2c-4c30-b06a-731c71077363/init-config-reloader/0.log" Jul 23 22:37:09.737067 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.737036 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zxb92_90109c89-0fc2-4084-b4c0-21f192e763cd/prometheus-operator/0.log" Jul 23 22:37:09.753736 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.753702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zxb92_90109c89-0fc2-4084-b4c0-21f192e763cd/kube-rbac-proxy/0.log" Jul 23 22:37:09.807637 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.807600 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6584c8969f-f6lc6_16f4ee53-5b4d-4c6c-b497-02d5ba76ee88/telemeter-client/0.log" Jul 23 22:37:09.828570 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.828544 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6584c8969f-f6lc6_16f4ee53-5b4d-4c6c-b497-02d5ba76ee88/reload/0.log" Jul 23 22:37:09.849697 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.849668 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6584c8969f-f6lc6_16f4ee53-5b4d-4c6c-b497-02d5ba76ee88/kube-rbac-proxy/0.log" Jul 23 22:37:09.879874 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.879792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/thanos-query/0.log" Jul 23 22:37:09.899469 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.899441 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/kube-rbac-proxy-web/0.log" Jul 23 22:37:09.928706 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.928669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/kube-rbac-proxy/0.log" Jul 23 22:37:09.948021 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.947987 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/prom-label-proxy/0.log" Jul 23 22:37:09.969096 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.969052 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/kube-rbac-proxy-rules/0.log" Jul 23 22:37:09.996192 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:09.996147 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887688fc8-7rt52_a38c4995-e680-4da3-b8b3-7f528b49d86a/kube-rbac-proxy-metrics/0.log" Jul 23 22:37:10.703118 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703080 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t"] Jul 23 22:37:10.703514 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703501 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703516 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703528 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703533 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703545 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703551 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703570 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703576 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703628 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703638 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5801076b-9446-4a26-9e17-24a6acc73db3" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703646 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60f1d3d-b030-460b-a1ed-ad89dd22a62b" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703651 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.703811 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.703658 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd7e8c3-bcc9-474b-895c-9732162d4da5" containerName="istio-proxy" Jul 23 22:37:10.706948 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.706928 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.709382 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.709354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wczc\"/\"kube-root-ca.crt\"" Jul 23 22:37:10.709382 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.709354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wczc\"/\"openshift-service-ca.crt\"" Jul 23 22:37:10.709572 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.709401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4wczc\"/\"default-dockercfg-bxs7g\"" Jul 23 22:37:10.716611 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.716581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t"] Jul 23 22:37:10.877351 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.877310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-proc\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.877351 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.877353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zs8\" (UniqueName: \"kubernetes.io/projected/50a079a2-0651-462e-b3cb-d16c43c8babc-kube-api-access-g2zs8\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.877619 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.877429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-lib-modules\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.877619 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.877464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-sys\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.877619 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.877524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-podres\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-proc\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zs8\" (UniqueName: \"kubernetes.io/projected/50a079a2-0651-462e-b3cb-d16c43c8babc-kube-api-access-g2zs8\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978172 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-lib-modules\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978379 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-sys\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978379 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978232 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-proc\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978379 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-podres\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978520 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-sys\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978520 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-lib-modules\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.978520 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.978457 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50a079a2-0651-462e-b3cb-d16c43c8babc-podres\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:10.986803 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:10.986784 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zs8\" (UniqueName: \"kubernetes.io/projected/50a079a2-0651-462e-b3cb-d16c43c8babc-kube-api-access-g2zs8\") pod \"perf-node-gather-daemonset-66z2t\" (UID: \"50a079a2-0651-462e-b3cb-d16c43c8babc\") " pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:11.017844 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:11.017819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:11.149886 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:11.149854 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t"] Jul 23 22:37:11.152049 ip-10-0-142-110 kubenswrapper[2569]: W0723 22:37:11.152012 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50a079a2_0651_462e_b3cb_d16c43c8babc.slice/crio-5ed5def53f0e5bc8571bc06c2084f262f06ba232289537186982a66ea8820b8a WatchSource:0}: Error finding container 5ed5def53f0e5bc8571bc06c2084f262f06ba232289537186982a66ea8820b8a: Status 404 returned error can't find the container with id 5ed5def53f0e5bc8571bc06c2084f262f06ba232289537186982a66ea8820b8a Jul 23 22:37:12.018412 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:12.018374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" event={"ID":"50a079a2-0651-462e-b3cb-d16c43c8babc","Type":"ContainerStarted","Data":"f296d207a3ac2edf0c0d8e7bc1ff0a57750e9efbcdb2fbcf8cae02b39ba7b713"} Jul 23 22:37:12.018964 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:12.018436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" event={"ID":"50a079a2-0651-462e-b3cb-d16c43c8babc","Type":"ContainerStarted","Data":"5ed5def53f0e5bc8571bc06c2084f262f06ba232289537186982a66ea8820b8a"} Jul 23 22:37:12.018964 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:12.018535 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:12.038722 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:12.038657 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" podStartSLOduration=2.038639419 podStartE2EDuration="2.038639419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:37:12.037264168 +0000 UTC m=+2712.062597703" watchObservedRunningTime="2026-07-23 22:37:12.038639419 +0000 UTC m=+2712.063972955" Jul 23 22:37:13.908810 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:13.908778 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wzj8m_900f4bc2-eed0-4e0c-bbbd-72759ca182b5/dns/0.log" Jul 23 22:37:13.927613 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:13.927582 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wzj8m_900f4bc2-eed0-4e0c-bbbd-72759ca182b5/kube-rbac-proxy/0.log" Jul 23 22:37:13.949221 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:13.949194 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h86nk_b6c64ade-a26a-4e1e-b1eb-776447b72dc5/dns-node-resolver/0.log" Jul 23 22:37:14.583723 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:14.583693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f5t8n_ee851f90-0e5c-4feb-accf-1128ed368f40/node-ca/0.log" Jul 23 22:37:15.524319 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:15.524282 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l4d8d_b0e6693d-0015-41f2-bf37-b09cb5aa177e/istio-proxy/0.log" Jul 23 22:37:15.743180 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:15.743141 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvs82_8ede2f35-344b-43a1-94e3-b86f569bb891/discovery/0.log" Jul 23 22:37:15.818454 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:15.818351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/0.log" Jul 23 22:37:15.874613 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:15.874572 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/1.log" Jul 23 22:37:16.652914 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:16.652886 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjjh2_11cf054f-80b1-4b3b-8869-43358dc4a75f/serve-healthcheck-canary/0.log" Jul 23 22:37:17.233846 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:17.233813 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtmcg_3f47888e-ef28-45ba-821e-6275fb2d3390/kube-rbac-proxy/0.log" Jul 23 22:37:17.255133 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:17.255106 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtmcg_3f47888e-ef28-45ba-821e-6275fb2d3390/exporter/0.log" Jul 23 22:37:17.278316 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:17.278286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gtmcg_3f47888e-ef28-45ba-821e-6275fb2d3390/extractor/0.log" Jul 23 22:37:18.032807 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:18.032779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4wczc/perf-node-gather-daemonset-66z2t" Jul 23 22:37:20.173523 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:20.173469 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp_35547a13-180d-475d-9a6e-934a0090328a/manager/0.log" Jul 23 22:37:20.192146 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:20.192115 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7bc97c8c-b2lxp_35547a13-180d-475d-9a6e-934a0090328a/copy-manifests/0.log" Jul 23 22:37:21.565460 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:21.565426 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66b46854f4-vhp92_8557a17c-e754-40f7-95af-8aeec255647b/manager/0.log" Jul 23 22:37:21.593422 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:21.593383 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dkb4h_97d2b7df-c98b-434d-bb35-292e89ec4cc6/openshift-lws-operator/0.log" Jul 23 22:37:21.998975 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:21.998944 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l4d8d_b0e6693d-0015-41f2-bf37-b09cb5aa177e/istio-proxy/0.log" Jul 23 22:37:22.218692 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:22.218659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvs82_8ede2f35-344b-43a1-94e3-b86f569bb891/discovery/0.log" Jul 23 22:37:22.270700 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:22.270599 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/0.log" Jul 23 22:37:22.326657 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:22.326624 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/1.log" Jul 23 22:37:23.529911 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:23.529881 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l4d8d_b0e6693d-0015-41f2-bf37-b09cb5aa177e/istio-proxy/0.log" Jul 23 22:37:23.744723 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:23.744689 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvs82_8ede2f35-344b-43a1-94e3-b86f569bb891/discovery/0.log" Jul 23 22:37:23.801105 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:23.801025 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/0.log" Jul 23 22:37:23.850033 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:23.850005 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-mltxm_90f47074-5175-4c4b-9398-8b5f7ab490b8/istio-proxy/1.log" Jul 23 22:37:28.107194 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.107154 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/kube-multus-additional-cni-plugins/0.log" Jul 23 22:37:28.126323 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.126293 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/egress-router-binary-copy/0.log" Jul 23 22:37:28.146214 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.146183 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/cni-plugins/0.log" Jul 23 22:37:28.170530 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.170473 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/bond-cni-plugin/0.log" Jul 23 22:37:28.191554 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.191527 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/routeoverride-cni/0.log" Jul 23 22:37:28.212682 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.212656 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/whereabouts-cni-bincopy/0.log" Jul 23 22:37:28.234259 ip-10-0-142-110 kubenswrapper[2569]: I0723 22:37:28.234232 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h5km9_439b24d5-9bad-4c69-9e1c-4bb7c94631d0/whereabouts-cni/0.log"