Jul 16 11:08:36.112704 ip-10-0-130-16 systemd[1]: Starting Kubernetes Kubelet... Jul 16 11:08:36.517468 ip-10-0-130-16 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:08:36.517468 ip-10-0-130-16 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 11:08:36.517468 ip-10-0-130-16 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:08:36.517468 ip-10-0-130-16 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 11:08:36.517468 ip-10-0-130-16 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 11:08:36.518112 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.517503 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 11:08:36.524579 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524557 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.524579 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524577 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.524579 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524581 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524585 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524588 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524592 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524595 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524599 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524603 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524607 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524612 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524615 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524618 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524621 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524624 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524627 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524630 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524632 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524636 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524638 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524641 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524644 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.524695 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524646 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524649 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524651 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524654 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524657 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524659 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524661 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524664 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524666 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524669 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524672 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524675 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524677 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524680 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524683 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524686 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524688 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524691 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524696 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524699 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.525161 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524702 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524705 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524708 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524711 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524713 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524716 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524718 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524721 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524723 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524729 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524733 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524736 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524738 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524741 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524744 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524746 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524749 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524751 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524754 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.525691 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524758 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524761 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524764 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524767 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524770 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524772 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524775 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524779 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524781 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524784 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524787 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524790 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524793 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524797 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524800 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524803 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524806 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524808 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524811 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524813 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.526135 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524816 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524819 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524821 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524824 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524827 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524829 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524831 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524834 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524837 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524839 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524842 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524845 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524849 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524853 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524856 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524858 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524861 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.526615 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.524864 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526919 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526927 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526931 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526934 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526937 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526940 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526943 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526946 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526949 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526953 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526955 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526958 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526961 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526964 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526966 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526969 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526972 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526974 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526977 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.527014 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526980 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526983 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526985 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526988 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526990 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526993 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526996 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.526998 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527001 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527004 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527006 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527009 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527011 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527014 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527018 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527021 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527024 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527027 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527030 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527032 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.527533 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527035 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527038 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527042 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527047 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527049 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527052 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527055 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527057 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527059 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527062 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527067 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527079 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527083 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527086 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527089 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527092 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527095 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527098 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527101 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.528235 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527104 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527107 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527109 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527112 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527115 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527120 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527123 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527126 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527128 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527131 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527134 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527136 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527139 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527144 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527148 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527150 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527153 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527155 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527158 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.528896 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527161 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527163 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527167 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527170 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527173 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527176 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527178 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527182 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527185 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527187 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527190 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527193 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527195 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527198 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527200 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527203 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527205 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527208 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527210 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.529400 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527214 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527217 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527940 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527954 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527962 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527967 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527972 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527983 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527989 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.527995 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528001 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528006 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528011 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528015 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528020 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528024 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528028 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528032 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528037 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528046 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.529854 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528051 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528056 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528060 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528065 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528069 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528073 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528077 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528080 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528084 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528089 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528093 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528097 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528106 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528110 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528115 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528120 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528123 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528128 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528132 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528136 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.530318 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528140 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528146 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528150 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528158 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528167 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528171 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528176 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528180 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528184 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528188 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528192 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528197 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528200 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528205 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528209 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528213 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528217 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528226 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528233 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.530802 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528238 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528244 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528249 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528254 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528258 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528263 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528268 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528272 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528276 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528280 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528289 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528294 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528298 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528302 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528308 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528315 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528321 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528327 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528332 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528336 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.531271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528340 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528349 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528353 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528357 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528362 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528366 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528370 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528391 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528397 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528400 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528405 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528409 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528413 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528417 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528426 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528430 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528435 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528440 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528445 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528449 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.531755 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528453 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.528458 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528614 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528625 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528636 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528643 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528650 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528661 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528670 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528677 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528682 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528687 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528693 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528699 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528704 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528714 2580 flags.go:64] FLAG: --cgroup-root="" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528720 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528726 2580 flags.go:64] FLAG: --client-ca-file="" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528731 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528736 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528742 2580 flags.go:64] FLAG: --cluster-domain="" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528748 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528785 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528839 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 11:08:36.532220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528844 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528959 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528964 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528968 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528972 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528975 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528978 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528982 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528985 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.528988 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529000 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529003 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529006 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529009 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529012 2580 flags.go:64] FLAG: --enable-server="true" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529015 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529022 2580 flags.go:64] FLAG: --event-burst="100" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529025 2580 flags.go:64] FLAG: --event-qps="50" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529028 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529031 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529034 2580 flags.go:64] FLAG: --eviction-hard="" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529038 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529041 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529044 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529048 2580 flags.go:64] FLAG: --eviction-soft="" Jul 16 11:08:36.532800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529051 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529054 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529058 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529061 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529063 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529066 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529070 2580 flags.go:64] FLAG: --feature-gates="" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529074 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529077 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529080 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529083 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529087 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529090 2580 flags.go:64] FLAG: --help="false" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529092 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529096 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529099 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529102 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529106 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529109 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529113 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529115 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529118 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529121 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 11:08:36.533407 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529124 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529128 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529130 2580 flags.go:64] FLAG: --kube-reserved="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529133 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529136 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529139 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529142 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529144 2580 flags.go:64] FLAG: --lock-file="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529148 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529151 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529154 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529159 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529162 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529165 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529168 2580 flags.go:64] FLAG: --logging-format="text" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529171 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529175 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529177 2580 flags.go:64] FLAG: --manifest-url="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529180 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529185 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529188 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529193 2580 flags.go:64] FLAG: --max-pods="110" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529196 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529199 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529202 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 11:08:36.534033 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529205 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529208 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529211 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529214 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529221 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529224 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529227 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529231 2580 flags.go:64] FLAG: --pod-cidr="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529234 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529240 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529243 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529247 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529249 2580 flags.go:64] FLAG: --port="10250" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529253 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529259 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fdfb5f44cf4ff1ef" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529262 2580 flags.go:64] FLAG: --qos-reserved="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529265 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529268 2580 flags.go:64] FLAG: --register-node="true" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529271 2580 flags.go:64] FLAG: --register-with-taints="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529274 2580 flags.go:64] FLAG: --registry-burst="10" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529277 2580 flags.go:64] FLAG: --registry-qps="5" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529280 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529283 2580 flags.go:64] FLAG: --reserved-memory="" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529287 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529290 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 11:08:36.534635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529293 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529296 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529299 2580 flags.go:64] FLAG: --runonce="false" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529302 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529305 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529308 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529311 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529313 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529316 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529319 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529322 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529325 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529328 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529331 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529334 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529337 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529339 2580 flags.go:64] FLAG: --system-cgroups="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529342 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529348 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529351 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529354 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529360 2580 flags.go:64] FLAG: --tls-min-version="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529363 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529365 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529368 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 11:08:36.535240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529371 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529391 2580 flags.go:64] FLAG: --v="2" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529396 2580 flags.go:64] FLAG: --version="false" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529401 2580 flags.go:64] FLAG: --vmodule="" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529406 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.529410 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529530 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529534 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529537 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529540 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529542 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529545 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529547 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529550 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529552 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529555 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529557 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529560 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529563 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529565 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529567 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.535858 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529571 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529575 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529578 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529581 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529584 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529588 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529590 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529594 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529597 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529599 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529602 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529604 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529607 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529609 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529612 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529614 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529618 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529620 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529623 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529626 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.536389 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529628 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529631 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529634 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529636 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529639 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529642 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529644 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529647 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529649 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529652 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529655 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529657 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529660 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529663 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529667 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529670 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529672 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529675 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529677 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529680 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.536875 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529683 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529686 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529688 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529691 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529693 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529696 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529698 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529701 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529704 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529708 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529710 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529713 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529715 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529718 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529720 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529723 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529725 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529728 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529731 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529733 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.537423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529736 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529738 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529741 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529743 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529746 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529749 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529751 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529754 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529757 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529759 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529762 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529765 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529767 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529771 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529775 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529778 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529781 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529784 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529787 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.537908 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529789 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529793 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529796 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529799 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.529802 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.530431 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.536982 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.536999 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.537013 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.537019 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537094 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537100 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537103 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537106 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.538360 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537109 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537112 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537115 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537118 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537121 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537124 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537126 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537129 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537132 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537135 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537138 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537140 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537143 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537146 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537149 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537151 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537154 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537156 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537159 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537162 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.538790 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537165 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537169 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537171 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537174 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537177 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537180 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537183 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537187 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537191 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537194 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537197 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537200 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537203 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537207 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537211 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537214 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537216 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537219 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537222 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.539268 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537225 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537227 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537230 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537233 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537235 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537238 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537241 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537243 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537246 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537249 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537252 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537255 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537258 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537260 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537263 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537266 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537268 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537272 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537275 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537277 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.539836 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537280 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537283 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537286 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537288 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537291 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537294 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537296 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537299 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537301 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537304 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537306 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537309 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537311 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537314 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537316 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537319 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537321 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537325 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537329 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537332 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.540330 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537334 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537337 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537340 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537342 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537345 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537347 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537350 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537353 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537356 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537359 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537362 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537365 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537368 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537370 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537390 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.540838 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537395 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.537400 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537523 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537529 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537532 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537534 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537537 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537540 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537543 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537545 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537548 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537550 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537553 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537556 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537558 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 11:08:36.541202 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537561 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537564 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537567 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537569 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537572 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537575 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537577 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537580 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537583 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537586 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537588 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537591 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537594 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537599 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537602 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537605 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537608 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537611 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537614 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 11:08:36.541596 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537617 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537620 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537623 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537626 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537628 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537631 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537634 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537636 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537640 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537643 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537646 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537649 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537652 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537654 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537657 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537659 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537662 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537666 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537669 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 11:08:36.542046 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537672 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537675 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537678 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537681 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537683 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537686 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537689 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537693 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537695 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537698 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537701 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537703 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537706 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537709 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537711 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537714 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537716 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537719 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537721 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537725 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 11:08:36.542557 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537727 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537730 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537732 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537735 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537737 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537740 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537742 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537745 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537748 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537750 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537753 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537755 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537758 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537761 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537763 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537767 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537769 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537772 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537774 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537778 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 11:08:36.543045 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537780 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537783 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537786 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537788 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537791 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537793 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537796 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:36.537798 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.537803 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.537928 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.540244 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.541662 2580 server.go:1019] "Starting client certificate rotation" Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.541767 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 11:08:36.543543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.541795 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 11:08:36.544525 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.544511 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 11:08:36.545972 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.545956 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 11:08:36.561311 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.561287 2580 log.go:25] "Validated CRI v1 runtime API" Jul 16 11:08:36.563760 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.563743 2580 log.go:25] "Validated CRI v1 image API" Jul 16 11:08:36.564692 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.564664 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 11:08:36.568703 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.568675 2580 fs.go:135] Filesystem UUIDs: map[53cb4cd8-652e-4ed9-9114-8903cbb9882f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80785847-6147-4e0c-b290-e89402eabb38:/dev/nvme0n1p4] Jul 16 11:08:36.568767 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.568703 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 11:08:36.575193 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.575067 2580 manager.go:216] Machine: {Timestamp:2026-07-16 11:08:36.573833369 +0000 UTC m=+0.339966496 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100210 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a7793569bd1486c203d59810be91 SystemUUID:ec28a779-3569-bd14-86c2-03d59810be91 BootID:60b688c7-4d8c-4b84-a681-025b1b0289bf Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d0:d6:98:88:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d0:d6:98:88:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:1a:87:b1:1f:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 11:08:36.575193 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.575180 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 11:08:36.575323 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.575269 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 11:08:36.576110 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.576094 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 11:08:36.578794 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.578752 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 11:08:36.578932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.578797 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 11:08:36.580098 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.580086 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 11:08:36.580139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.580100 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 11:08:36.580139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.580108 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 11:08:36.581404 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.581393 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 11:08:36.582647 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.582636 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 11:08:36.586190 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586178 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 16 11:08:36.586311 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586302 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 11:08:36.586344 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586333 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 16 11:08:36.586388 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586345 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 11:08:36.586388 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586359 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 11:08:36.586388 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586370 2580 kubelet.go:398] "Adding apiserver pod source" Jul 16 11:08:36.586471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586395 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 11:08:36.586882 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586866 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 11:08:36.586945 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.586891 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 11:08:36.589178 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.589159 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 11:08:36.591116 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591102 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 11:08:36.591177 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591132 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 11:08:36.591299 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591289 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 11:08:36.591340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591303 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 11:08:36.591340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591309 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 11:08:36.591340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591320 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 11:08:36.591340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591328 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 11:08:36.591340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591333 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 11:08:36.591784 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591775 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 11:08:36.591784 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591784 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 11:08:36.591844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591791 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 11:08:36.591844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591797 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 11:08:36.591844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591805 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 11:08:36.591844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.591813 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 11:08:36.592573 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.592561 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 11:08:36.592573 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.592571 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 11:08:36.595942 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.595928 2580 server.go:1268] "Started kubelet" Jul 16 11:08:36.596046 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.596009 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 11:08:36.596172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.596147 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 11:08:36.596612 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.596591 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 11:08:36.596693 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.596603 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 11:08:36.596693 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.596627 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 11:08:36.596693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.596674 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 11:08:36.597023 ip-10-0-130-16 systemd[1]: Started Kubernetes Kubelet. Jul 16 11:08:36.597393 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.597358 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 11:08:36.597693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.597568 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zm576" Jul 16 11:08:36.599188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.599168 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 16 11:08:36.604081 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.604057 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 11:08:36.604293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.604280 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 11:08:36.604403 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.604361 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 11:08:36.604524 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.604500 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:36.604976 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.604124 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-16.ec2.internal.18c2c08d8de8b899 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-16.ec2.internal,UID:ip-10-0-130-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-16.ec2.internal,},FirstTimestamp:2026-07-16 11:08:36.595898521 +0000 UTC m=+0.362031649,LastTimestamp:2026-07-16 11:08:36.595898521 +0000 UTC m=+0.362031649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-16.ec2.internal,}" Jul 16 11:08:36.604976 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.604947 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 11:08:36.605139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605085 2580 factory.go:55] Registering systemd factory Jul 16 11:08:36.605139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605112 2580 factory.go:223] Registration of the systemd container factory successfully Jul 16 11:08:36.605223 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605179 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 11:08:36.605223 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605190 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 11:08:36.605310 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605249 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 16 11:08:36.605310 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605257 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 16 11:08:36.605418 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605355 2580 factory.go:153] Registering CRI-O factory Jul 16 11:08:36.605487 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605370 2580 factory.go:223] Registration of the crio container factory successfully Jul 16 11:08:36.605563 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605549 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 11:08:36.605614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605580 2580 factory.go:103] Registering Raw factory Jul 16 11:08:36.605614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605596 2580 manager.go:1196] Started watching for new ooms in manager Jul 16 11:08:36.605961 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.605947 2580 manager.go:349] Starting recovery of all containers Jul 16 11:08:36.607322 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.607299 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zm576" Jul 16 11:08:36.610037 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.609934 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 11:08:36.610037 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.609963 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 11:08:36.619592 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.619423 2580 manager.go:354] Recovery completed Jul 16 11:08:36.620841 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.620805 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 16 11:08:36.624067 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.624055 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.626114 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626091 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.626206 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626131 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.626206 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626145 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.626755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626740 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 11:08:36.626755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626754 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 11:08:36.626876 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626773 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 16 11:08:36.626876 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.626752 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.627514 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627496 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.627589 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627526 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.627589 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627540 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.627936 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627921 2580 policy_none.go:49] "None policy: Start" Jul 16 11:08:36.627936 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627938 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 11:08:36.628038 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.627949 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 11:08:36.629247 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.629230 2580 policy_none.go:47] "Start" Jul 16 11:08:36.670058 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670041 2580 manager.go:337] "Starting Device Plugin manager" Jul 16 11:08:36.670143 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.670096 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 11:08:36.670143 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670112 2580 server.go:87] "Starting device plugin registration server" Jul 16 11:08:36.670419 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670404 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 11:08:36.670542 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670419 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 11:08:36.670542 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670491 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 11:08:36.670635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670579 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 11:08:36.670635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.670586 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 11:08:36.671175 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.671145 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 11:08:36.671270 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.671193 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:36.770847 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.770775 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.772039 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.771919 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.772039 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.771949 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.772039 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.771958 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.772039 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.771985 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.779988 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.779970 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.780040 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.779994 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-16.ec2.internal\": node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:36.794952 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.794921 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 11:08:36.796093 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.796064 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 11:08:36.796182 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.796115 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 11:08:36.796182 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.796139 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 11:08:36.796257 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.796181 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 11:08:36.798402 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.798365 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:36.798883 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.798867 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:08:36.897102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.897062 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal"] Jul 16 11:08:36.897264 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.897145 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.898539 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.898522 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:36.898878 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.898863 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.898930 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.898901 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.898930 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.898918 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.900654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.900642 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.900813 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.900801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.900857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.900828 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.901368 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901354 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.901452 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901432 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.901504 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901455 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.901504 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901469 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.901808 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901795 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.901859 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.901816 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.902665 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.902649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.902744 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.902681 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 11:08:36.903329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.903314 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientMemory" Jul 16 11:08:36.903395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.903340 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 11:08:36.903395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.903351 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeHasSufficientPID" Jul 16 11:08:36.906148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.906132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.906222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.906155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.906222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:36.906173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/275e2b67084dfc9c3fa0f7611e1a99f6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-16.ec2.internal\" (UID: \"275e2b67084dfc9c3fa0f7611e1a99f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.927048 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.927023 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-16.ec2.internal\" not found" node="ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.931510 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.931492 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-16.ec2.internal\" not found" node="ip-10-0-130-16.ec2.internal" Jul 16 11:08:36.999431 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:36.999401 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.006731 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.006806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/275e2b67084dfc9c3fa0f7611e1a99f6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-16.ec2.internal\" (UID: \"275e2b67084dfc9c3fa0f7611e1a99f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.006806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.006806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.006928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04e9f6389d0c6d59e071fd815e191794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal\" (UID: \"04e9f6389d0c6d59e071fd815e191794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.006928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.006846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/275e2b67084dfc9c3fa0f7611e1a99f6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-16.ec2.internal\" (UID: \"275e2b67084dfc9c3fa0f7611e1a99f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.100205 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.100126 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.200625 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.200598 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.229159 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.229125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.235056 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.235035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.301732 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.301694 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.402248 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.402176 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.502707 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.502676 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.541219 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.541182 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 11:08:37.541805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.541338 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 11:08:37.603254 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.603226 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.611695 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.611670 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 11:03:36 +0000 UTC" deadline="2028-02-06 03:47:13.352651799 +0000 UTC" Jul 16 11:08:37.611816 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.611695 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13672h38m35.740959243s" Jul 16 11:08:37.703795 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:37.703761 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-16.ec2.internal\" not found" Jul 16 11:08:37.706286 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:37.706253 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e9f6389d0c6d59e071fd815e191794.slice/crio-41e5fd0ebb463657acd3d22ea61b79eb236199d729ccceb988b10d7b55156504 WatchSource:0}: Error finding container 41e5fd0ebb463657acd3d22ea61b79eb236199d729ccceb988b10d7b55156504: Status 404 returned error can't find the container with id 41e5fd0ebb463657acd3d22ea61b79eb236199d729ccceb988b10d7b55156504 Jul 16 11:08:37.706884 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:37.706862 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275e2b67084dfc9c3fa0f7611e1a99f6.slice/crio-7264217013b9df33005208492b00655a2c3e33f3457e1dd0d079b63b791aa539 WatchSource:0}: Error finding container 7264217013b9df33005208492b00655a2c3e33f3457e1dd0d079b63b791aa539: Status 404 returned error can't find the container with id 7264217013b9df33005208492b00655a2c3e33f3457e1dd0d079b63b791aa539 Jul 16 11:08:37.713191 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.713172 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:08:37.716525 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.716506 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:08:37.799201 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.799145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" event={"ID":"275e2b67084dfc9c3fa0f7611e1a99f6","Type":"ContainerStarted","Data":"7264217013b9df33005208492b00655a2c3e33f3457e1dd0d079b63b791aa539"} Jul 16 11:08:37.800192 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.800171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" event={"ID":"04e9f6389d0c6d59e071fd815e191794","Type":"ContainerStarted","Data":"41e5fd0ebb463657acd3d22ea61b79eb236199d729ccceb988b10d7b55156504"} Jul 16 11:08:37.804496 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.804476 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.807472 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.807455 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:08:37.814062 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.814047 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 11:08:37.815022 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.815010 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" Jul 16 11:08:37.821504 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:37.821489 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 11:08:38.077597 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.077517 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:08:38.588053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.588024 2580 apiserver.go:52] "Watching apiserver" Jul 16 11:08:38.594285 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.594261 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 11:08:38.595479 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.595447 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jkddl","kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal","openshift-cluster-node-tuning-operator/tuned-c5s45","openshift-dns/node-resolver-d6gc4","openshift-image-registry/node-ca-js4ck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal","openshift-multus/network-metrics-daemon-fkqxh","openshift-network-diagnostics/network-check-target-bhbv2","kube-system/konnectivity-agent-szwbp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld","openshift-multus/multus-additional-cni-plugins-8npjw","openshift-multus/multus-fkmbq","openshift-network-operator/iptables-alerter-ll285","openshift-ovn-kubernetes/ovnkube-node-vrmqc"] Jul 16 11:08:38.597123 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.597100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.597210 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.597176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:38.598468 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.598451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.599691 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.599671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.600751 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.600682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 11:08:38.600751 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.600707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.600751 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.600723 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.600751 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.600742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpwc\"" Jul 16 11:08:38.600970 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.600945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.601651 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.601632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2hxt\"" Jul 16 11:08:38.601741 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.601635 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.601741 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.601722 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.602230 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.602211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.602798 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.602780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-szdqr\"" Jul 16 11:08:38.602963 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.602942 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.603052 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.603038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.603677 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.603661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.604891 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.605020 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.605219 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.605497 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.605798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2ddm\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606055 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606557 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdkpw\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606590 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606747 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606759 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 11:08:38.606982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.606879 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 11:08:38.607855 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.607645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.609678 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.609656 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 11:08:38.610199 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.609876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 11:08:38.610199 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.609915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.610199 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.609949 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jxmcr\"" Jul 16 11:08:38.610199 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.609989 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:38.611671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.611651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:38.611764 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.611723 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:38.613027 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.613839 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-conf\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.613938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-run\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.613938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.613938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9qv\" (UniqueName: \"kubernetes.io/projected/292401f4-8e04-4b7d-b3dc-0604062897e4-kube-api-access-jz9qv\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.613938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-sys\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75qw\" (UniqueName: \"kubernetes.io/projected/269c6854-9df1-4fb2-a189-c2d56701199e-kube-api-access-x75qw\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.613968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-systemd-units\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-slash\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-script-lib\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpj5\" (UniqueName: \"kubernetes.io/projected/8cecad5f-d289-4c00-b126-035f040784b8-kube-api-access-7tpj5\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-modprobe-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78778387-11b8-464f-8f4f-b791c7e46005-tmp-dir\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-var-lib-kubelet\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-etc-tuned\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-netns\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-var-lib-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-bin\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-lib-modules\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-host\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-etc-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-netd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-config\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.614509 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/292401f4-8e04-4b7d-b3dc-0604062897e4-host-slash\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78778387-11b8-464f-8f4f-b791c7e46005-hosts-file\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdfn\" (UniqueName: \"kubernetes.io/projected/78778387-11b8-464f-8f4f-b791c7e46005-kube-api-access-wvdfn\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysconfig\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-systemd\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/269c6854-9df1-4fb2-a189-c2d56701199e-serviceca\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/444e123d-6ada-4f32-8754-d805cb14097a-agent-certs\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-kubernetes\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-ovn\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-node-log\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-env-overrides\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklj9\" (UniqueName: \"kubernetes.io/projected/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-kube-api-access-tklj9\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/269c6854-9df1-4fb2-a189-c2d56701199e-host\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.614982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/444e123d-6ada-4f32-8754-d805cb14097a-konnectivity-ca\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5dr\" (UniqueName: \"kubernetes.io/projected/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-kube-api-access-xk5dr\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.615105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/292401f4-8e04-4b7d-b3dc-0604062897e4-iptables-alerter-script\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-tmp\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-systemd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-log-socket\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-kubelet\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovn-node-metrics-cert\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615355 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kxxcz\"" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615586 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 11:08:38.615927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.615876 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 11:08:38.616445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.616110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.616554 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.616538 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 11:08:38.616688 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.616659 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 11:08:38.616834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.616818 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 11:08:38.616926 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.616911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 11:08:38.617224 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.617206 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 11:08:38.617486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.617403 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m6rlw\"" Jul 16 11:08:38.617974 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.617955 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 11:08:38.618064 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.617992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fvlq9\"" Jul 16 11:08:38.635972 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.635952 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hs4rh" Jul 16 11:08:38.643876 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.643856 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hs4rh" Jul 16 11:08:38.706264 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.706228 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 11:08:38.715667 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-ovn\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/444e123d-6ada-4f32-8754-d805cb14097a-agent-certs\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-kubernetes\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-hostroot\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-ovn\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-multus-certs\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-kubernetes\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.715806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-etc-kubernetes\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/269c6854-9df1-4fb2-a189-c2d56701199e-host\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2qp\" (UniqueName: \"kubernetes.io/projected/42c6d016-3953-49a4-b2d0-3268fba259dd-kube-api-access-bk2qp\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-kubelet\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/269c6854-9df1-4fb2-a189-c2d56701199e-host\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-kubelet\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-conf\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-k8s-cni-cncf-io\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.715998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-slash\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-conf\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cnibin\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-kubelet\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716064 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 11:08:38.716120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78778387-11b8-464f-8f4f-b791c7e46005-tmp-dir\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-var-lib-kubelet\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-var-lib-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.716160 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-binary-copy\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-var-lib-kubelet\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-config\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-slash\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-var-lib-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-system-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.716238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:39.21621601 +0000 UTC m=+2.982349147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-socket-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/292401f4-8e04-4b7d-b3dc-0604062897e4-host-slash\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdfn\" (UniqueName: \"kubernetes.io/projected/78778387-11b8-464f-8f4f-b791c7e46005-kube-api-access-wvdfn\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysconfig\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.716925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/292401f4-8e04-4b7d-b3dc-0604062897e4-host-slash\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78778387-11b8-464f-8f4f-b791c7e46005-tmp-dir\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-node-log\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-env-overrides\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysconfig\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklj9\" (UniqueName: \"kubernetes.io/projected/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-kube-api-access-tklj9\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-node-log\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmsm\" (UniqueName: \"kubernetes.io/projected/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-kube-api-access-fdmsm\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/444e123d-6ada-4f32-8754-d805cb14097a-konnectivity-ca\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5dr\" (UniqueName: \"kubernetes.io/projected/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-kube-api-access-xk5dr\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/292401f4-8e04-4b7d-b3dc-0604062897e4-iptables-alerter-script\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-tmp\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-systemd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-log-socket\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.717750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovn-node-metrics-cert\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-run-systemd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9mj\" (UniqueName: \"kubernetes.io/projected/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-kube-api-access-dn9mj\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-config\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-os-release\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cni-binary-copy\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-netns\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-bin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-log-socket\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-conf-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.716994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-run\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-env-overrides\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-sys-fs\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-kubelet-config\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-dbus\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-system-cni-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.718528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-multus\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-registration-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-run\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/444e123d-6ada-4f32-8754-d805cb14097a-konnectivity-ca\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9qv\" (UniqueName: \"kubernetes.io/projected/292401f4-8e04-4b7d-b3dc-0604062897e4-kube-api-access-jz9qv\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-sys\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75qw\" (UniqueName: \"kubernetes.io/projected/269c6854-9df1-4fb2-a189-c2d56701199e-kube-api-access-x75qw\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-systemd-units\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-sys\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-script-lib\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-daemon-config\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/292401f4-8e04-4b7d-b3dc-0604062897e4-iptables-alerter-script\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpj5\" (UniqueName: \"kubernetes.io/projected/8cecad5f-d289-4c00-b126-035f040784b8-kube-api-access-7tpj5\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-systemd-units\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-modprobe-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-etc-tuned\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-netns\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-bin\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-modprobe-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-netns\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cnibin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-bin\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-lib-modules\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-host\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovnkube-script-lib\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-etc-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-netd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-host\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-sysctl-d\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.719932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-os-release\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-host-cni-netd\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.717992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-socket-dir-parent\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-lib-modules\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-device-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-etc-openvswitch\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78778387-11b8-464f-8f4f-b791c7e46005-hosts-file\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-systemd\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78778387-11b8-464f-8f4f-b791c7e46005-hosts-file\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/269c6854-9df1-4fb2-a189-c2d56701199e-serviceca\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cecad5f-d289-4c00-b126-035f040784b8-etc-systemd\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.718719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/269c6854-9df1-4fb2-a189-c2d56701199e-serviceca\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.719737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-tmp\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.719827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-ovn-node-metrics-cert\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.719973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/444e123d-6ada-4f32-8754-d805cb14097a-agent-certs\") pod \"konnectivity-agent-szwbp\" (UID: \"444e123d-6ada-4f32-8754-d805cb14097a\") " pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.720466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.720273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cecad5f-d289-4c00-b126-035f040784b8-etc-tuned\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.724229 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.724149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklj9\" (UniqueName: \"kubernetes.io/projected/1c15c583-3e8a-4ea6-9cf4-45c3702d0517-kube-api-access-tklj9\") pod \"ovnkube-node-vrmqc\" (UID: \"1c15c583-3e8a-4ea6-9cf4-45c3702d0517\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.724575 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.724554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdfn\" (UniqueName: \"kubernetes.io/projected/78778387-11b8-464f-8f4f-b791c7e46005-kube-api-access-wvdfn\") pod \"node-resolver-d6gc4\" (UID: \"78778387-11b8-464f-8f4f-b791c7e46005\") " pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.724930 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.724907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75qw\" (UniqueName: \"kubernetes.io/projected/269c6854-9df1-4fb2-a189-c2d56701199e-kube-api-access-x75qw\") pod \"node-ca-js4ck\" (UID: \"269c6854-9df1-4fb2-a189-c2d56701199e\") " pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.725575 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.725559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9qv\" (UniqueName: \"kubernetes.io/projected/292401f4-8e04-4b7d-b3dc-0604062897e4-kube-api-access-jz9qv\") pod \"iptables-alerter-ll285\" (UID: \"292401f4-8e04-4b7d-b3dc-0604062897e4\") " pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.725643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.725574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5dr\" (UniqueName: \"kubernetes.io/projected/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-kube-api-access-xk5dr\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:38.725946 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.725930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpj5\" (UniqueName: \"kubernetes.io/projected/8cecad5f-d289-4c00-b126-035f040784b8-kube-api-access-7tpj5\") pod \"tuned-c5s45\" (UID: \"8cecad5f-d289-4c00-b126-035f040784b8\") " pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.818567 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-device-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-hostroot\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-multus-certs\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-etc-kubernetes\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2qp\" (UniqueName: \"kubernetes.io/projected/42c6d016-3953-49a4-b2d0-3268fba259dd-kube-api-access-bk2qp\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-device-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-multus-certs\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.818720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-k8s-cni-cncf-io\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-etc-kubernetes\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cnibin\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cnibin\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-kubelet\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-binary-copy\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-kubelet\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-hostroot\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-system-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-system-cni-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-k8s-cni-cncf-io\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.818984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-socket-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmsm\" (UniqueName: \"kubernetes.io/projected/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-kube-api-access-fdmsm\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-socket-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9mj\" (UniqueName: \"kubernetes.io/projected/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-kube-api-access-dn9mj\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-os-release\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cni-binary-copy\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-netns\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-bin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-conf-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-sys-fs\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-kubelet-config\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-dbus\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-conf-dir\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-system-cni-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-multus\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.819715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-registration-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-sys-fs\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-daemon-config\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-binary-copy\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-run-netns\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-multus\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-kubelet-config\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee647ad6-cd55-4717-b5ac-618f94505d75-dbus\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-host-var-lib-cni-bin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cni-binary-copy\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.819703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-os-release\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42c6d016-3953-49a4-b2d0-3268fba259dd-registration-dir\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-system-cni-dir\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:38.820462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cnibin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-daemon-config\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-os-release\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.820553 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-os-release\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-cnibin\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-socket-dir-parent\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.820818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-multus-socket-dir-parent\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.821254 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.820860 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:39.320835177 +0000 UTC m=+3.086968298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:38.821671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.821349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.824763 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.824743 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:38.824876 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.824773 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:38.824876 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.824788 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:38.824876 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:38.824849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:39.324832092 +0000 UTC m=+3.090965206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:38.827234 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.827208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9mj\" (UniqueName: \"kubernetes.io/projected/4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded-kube-api-access-dn9mj\") pod \"multus-additional-cni-plugins-8npjw\" (UID: \"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded\") " pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.827780 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.827739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2qp\" (UniqueName: \"kubernetes.io/projected/42c6d016-3953-49a4-b2d0-3268fba259dd-kube-api-access-bk2qp\") pod \"aws-ebs-csi-driver-node-g4vld\" (UID: \"42c6d016-3953-49a4-b2d0-3268fba259dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.827926 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.827906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmsm\" (UniqueName: \"kubernetes.io/projected/8b78c9be-e00d-4dbb-afaa-b9412a469b7e-kube-api-access-fdmsm\") pod \"multus-fkmbq\" (UID: \"8b78c9be-e00d-4dbb-afaa-b9412a469b7e\") " pod="openshift-multus/multus-fkmbq" Jul 16 11:08:38.845641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.845590 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 11:08:38.911439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.911403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ll285" Jul 16 11:08:38.919169 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.919144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5s45" Jul 16 11:08:38.927010 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.926982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d6gc4" Jul 16 11:08:38.932627 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.932608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-js4ck" Jul 16 11:08:38.939491 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.939472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:08:38.945189 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.945169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:08:38.951034 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.951016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" Jul 16 11:08:38.956699 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.956674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8npjw" Jul 16 11:08:38.960415 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:38.960395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fkmbq" Jul 16 11:08:39.223657 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.223585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:39.223811 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.223734 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:39.223811 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.223794 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:40.22377933 +0000 UTC m=+3.989912458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:39.324220 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.324177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:39.324386 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.324315 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:39.324439 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.324387 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:40.324360231 +0000 UTC m=+4.090493351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:39.424798 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.424763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:39.424961 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.424928 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:39.424961 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.424949 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:39.424961 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.424960 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:39.425083 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.425013 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:40.424999664 +0000 UTC m=+4.191132778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:39.442200 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.442169 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78778387_11b8_464f_8f4f_b791c7e46005.slice/crio-480a8b658ea1b01b0780fcdac0a39df6878ba56f1d7713941d0100f9b9971acc WatchSource:0}: Error finding container 480a8b658ea1b01b0780fcdac0a39df6878ba56f1d7713941d0100f9b9971acc: Status 404 returned error can't find the container with id 480a8b658ea1b01b0780fcdac0a39df6878ba56f1d7713941d0100f9b9971acc Jul 16 11:08:39.443916 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.443891 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292401f4_8e04_4b7d_b3dc_0604062897e4.slice/crio-4b21624d4fd9934b76afb85eed7a41ba1aa054d5a851e85f02fada9269897942 WatchSource:0}: Error finding container 4b21624d4fd9934b76afb85eed7a41ba1aa054d5a851e85f02fada9269897942: Status 404 returned error can't find the container with id 4b21624d4fd9934b76afb85eed7a41ba1aa054d5a851e85f02fada9269897942 Jul 16 11:08:39.445729 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.445626 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cecad5f_d289_4c00_b126_035f040784b8.slice/crio-ae8bb39eb188fc0391b347faf4c575900e9a4cffa528f9ee866ab3d5ca9fdfe9 WatchSource:0}: Error finding container ae8bb39eb188fc0391b347faf4c575900e9a4cffa528f9ee866ab3d5ca9fdfe9: Status 404 returned error can't find the container with id ae8bb39eb188fc0391b347faf4c575900e9a4cffa528f9ee866ab3d5ca9fdfe9 Jul 16 11:08:39.448351 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.448326 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444e123d_6ada_4f32_8754_d805cb14097a.slice/crio-1931ff6f345a8f5703fc78a8b3d463caabc64dbed8c7f75a0fd2182256e60dea WatchSource:0}: Error finding container 1931ff6f345a8f5703fc78a8b3d463caabc64dbed8c7f75a0fd2182256e60dea: Status 404 returned error can't find the container with id 1931ff6f345a8f5703fc78a8b3d463caabc64dbed8c7f75a0fd2182256e60dea Jul 16 11:08:39.449035 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.449014 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b78c9be_e00d_4dbb_afaa_b9412a469b7e.slice/crio-1a90e35686587e52739d04290d6b0b859ebc654f9987917a342418a5bd44dfea WatchSource:0}: Error finding container 1a90e35686587e52739d04290d6b0b859ebc654f9987917a342418a5bd44dfea: Status 404 returned error can't find the container with id 1a90e35686587e52739d04290d6b0b859ebc654f9987917a342418a5bd44dfea Jul 16 11:08:39.450411 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.450365 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfb2f7d_dfe9_4933_8d6a_ee97d8c7aded.slice/crio-3e85c18b44e83ec5e7a4eba0a8eb1f22a8535f24e8c358b331517f0e17761142 WatchSource:0}: Error finding container 3e85c18b44e83ec5e7a4eba0a8eb1f22a8535f24e8c358b331517f0e17761142: Status 404 returned error can't find the container with id 3e85c18b44e83ec5e7a4eba0a8eb1f22a8535f24e8c358b331517f0e17761142 Jul 16 11:08:39.451271 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:08:39.451249 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c15c583_3e8a_4ea6_9cf4_45c3702d0517.slice/crio-37605cb27d6a13ee2030df33471ad2e06336c33d9ccc4a9f8535f05f3c9df797 WatchSource:0}: Error finding container 37605cb27d6a13ee2030df33471ad2e06336c33d9ccc4a9f8535f05f3c9df797: Status 404 returned error can't find the container with id 37605cb27d6a13ee2030df33471ad2e06336c33d9ccc4a9f8535f05f3c9df797 Jul 16 11:08:39.646535 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.646297 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 11:03:38 +0000 UTC" deadline="2028-04-06 16:11:38.444051722 +0000 UTC" Jul 16 11:08:39.646535 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.646532 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15125h2m58.797524397s" Jul 16 11:08:39.797301 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.797205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:39.797500 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:39.797335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:39.804546 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.804506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d6gc4" event={"ID":"78778387-11b8-464f-8f4f-b791c7e46005","Type":"ContainerStarted","Data":"480a8b658ea1b01b0780fcdac0a39df6878ba56f1d7713941d0100f9b9971acc"} Jul 16 11:08:39.806633 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.806600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" event={"ID":"42c6d016-3953-49a4-b2d0-3268fba259dd","Type":"ContainerStarted","Data":"97c0b5c8169b72dbefa598de0adf4d049a82249dfbaa135a80b2d2530a669000"} Jul 16 11:08:39.808040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.808008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-js4ck" event={"ID":"269c6854-9df1-4fb2-a189-c2d56701199e","Type":"ContainerStarted","Data":"ccab46569aea28bad59c3b67c6e0d3b94fa10572e5391573b3b28f7a654e1dda"} Jul 16 11:08:39.809640 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.809615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"37605cb27d6a13ee2030df33471ad2e06336c33d9ccc4a9f8535f05f3c9df797"} Jul 16 11:08:39.811103 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.811060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szwbp" event={"ID":"444e123d-6ada-4f32-8754-d805cb14097a","Type":"ContainerStarted","Data":"1931ff6f345a8f5703fc78a8b3d463caabc64dbed8c7f75a0fd2182256e60dea"} Jul 16 11:08:39.813876 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.813490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" event={"ID":"275e2b67084dfc9c3fa0f7611e1a99f6","Type":"ContainerStarted","Data":"c9b002c608c9189b9e7b3f0f3c6ca92c6c82a9b020c82c630113a62311b2d77f"} Jul 16 11:08:39.814882 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.814846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerStarted","Data":"3e85c18b44e83ec5e7a4eba0a8eb1f22a8535f24e8c358b331517f0e17761142"} Jul 16 11:08:39.816867 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.816765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fkmbq" event={"ID":"8b78c9be-e00d-4dbb-afaa-b9412a469b7e","Type":"ContainerStarted","Data":"1a90e35686587e52739d04290d6b0b859ebc654f9987917a342418a5bd44dfea"} Jul 16 11:08:39.818169 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.818147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5s45" event={"ID":"8cecad5f-d289-4c00-b126-035f040784b8","Type":"ContainerStarted","Data":"ae8bb39eb188fc0391b347faf4c575900e9a4cffa528f9ee866ab3d5ca9fdfe9"} Jul 16 11:08:39.819413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.819390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ll285" event={"ID":"292401f4-8e04-4b7d-b3dc-0604062897e4","Type":"ContainerStarted","Data":"4b21624d4fd9934b76afb85eed7a41ba1aa054d5a851e85f02fada9269897942"} Jul 16 11:08:39.827354 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:39.827301 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-16.ec2.internal" podStartSLOduration=2.82728766 podStartE2EDuration="2.82728766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:39.826697566 +0000 UTC m=+3.592830709" watchObservedRunningTime="2026-07-16 11:08:39.82728766 +0000 UTC m=+3.593420798" Jul 16 11:08:40.230821 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.230647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:40.231018 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.230821 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:40.231018 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.230905 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:42.230882449 +0000 UTC m=+5.997015570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:40.332177 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.331567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:40.332177 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.331766 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:40.332177 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.331836 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:42.331817545 +0000 UTC m=+6.097950663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:40.432627 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.432530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:40.432820 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.432689 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:40.432820 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.432711 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:40.432820 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.432724 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:40.432820 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.432781 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:42.432764411 +0000 UTC m=+6.198897540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:40.647023 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.646937 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 11:03:38 +0000 UTC" deadline="2028-03-23 03:21:21.130868408 +0000 UTC" Jul 16 11:08:40.647023 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.647019 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14776h12m40.483853853s" Jul 16 11:08:40.797602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.797521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:40.797772 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.797640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:40.798154 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:40.798070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:40.798229 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:40.798165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:41.798023 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:41.797240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:41.798023 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:41.797430 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:42.247125 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:42.247033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:42.247310 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.247235 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:42.247310 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.247300 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:46.247281026 +0000 UTC m=+10.013414144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:42.348256 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:42.347634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:42.348256 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.347798 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:42.348256 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.347864 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:46.347845206 +0000 UTC m=+10.113978324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:42.448925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:42.448886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:42.449166 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.449112 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:42.449166 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.449139 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:42.449166 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.449153 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:42.449351 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.449249 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:46.449228962 +0000 UTC m=+10.215362083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:42.797686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:42.797031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:42.797686 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.797167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:42.797686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:42.797551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:42.797686 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:42.797639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:43.796639 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:43.796605 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:43.797099 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:43.796753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:44.796639 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:44.796603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:44.797119 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:44.796601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:44.797119 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:44.796746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:44.797119 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:44.796812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:45.797867 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:45.797368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:45.797867 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:45.797535 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:46.283818 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:46.283680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:46.283818 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.283822 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:46.284072 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.283895 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:54.283873562 +0000 UTC m=+18.050006680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:46.384143 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:46.384077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:46.384314 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.384232 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:46.384314 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.384294 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:54.384275536 +0000 UTC m=+18.150408658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:46.484454 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:46.484420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:46.484643 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.484588 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:46.484643 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.484609 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:46.484643 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.484622 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:46.484798 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.484688 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:08:54.484667781 +0000 UTC m=+18.250800903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:46.797838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:46.797507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:46.797838 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.797621 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:46.797838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:46.797711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:46.797838 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:46.797798 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:47.796856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:47.796774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:47.797009 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:47.796913 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:48.796490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:48.796450 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:48.796490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:48.796478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:48.797048 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:48.796582 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:48.797048 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:48.796711 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:49.797318 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:49.797275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:49.797742 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:49.797419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:50.797147 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:50.797109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:50.797313 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:50.797117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:50.797313 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:50.797247 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:50.797736 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:50.797339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:51.797015 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:51.796976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:51.797166 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:51.797112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:52.796642 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:52.796603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:52.796642 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:52.796640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:52.797113 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:52.796732 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:52.797113 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:52.796842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:53.797298 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:53.797263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:53.797822 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:53.797418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:54.346504 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:54.346462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:54.346673 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.346637 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:54.346726 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.346707 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:10.346687984 +0000 UTC m=+34.112821104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:08:54.447831 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:54.447789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:54.448000 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.447959 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:54.448055 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.448027 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:10.448008087 +0000 UTC m=+34.214141201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:08:54.548945 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:54.548900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:54.549130 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.549108 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:08:54.549177 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.549142 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:08:54.549177 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.549156 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:54.549262 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.549225 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:10.549206671 +0000 UTC m=+34.315339801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:08:54.797249 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:54.797202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:54.797448 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.797336 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:54.797448 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:54.797405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:54.797871 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:54.797495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:55.796390 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:55.796349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:55.796549 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:55.796487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:56.797162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:56.797123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:56.797602 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:56.797227 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:56.797602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:56.797279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:56.797602 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:56.797419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:57.797007 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:57.796968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:57.797180 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:57.797102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:58.797099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.796856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:08:58.797217 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.796920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:08:58.797515 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:58.797220 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:08:58.797515 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:58.797288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:08:58.864578 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.864540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fkmbq" event={"ID":"8b78c9be-e00d-4dbb-afaa-b9412a469b7e","Type":"ContainerStarted","Data":"c0f7955226f9848ed1303b7bf40e875947da481485b35c058eb3b761e2a84cb0"} Jul 16 11:08:58.866656 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.866621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5s45" event={"ID":"8cecad5f-d289-4c00-b126-035f040784b8","Type":"ContainerStarted","Data":"40fed296605e6fa8fdcf343b5ebca468084adeb2765a9157a60ebf24cea6e6ee"} Jul 16 11:08:58.883634 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.883570 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fkmbq" podStartSLOduration=3.966080751 podStartE2EDuration="22.883550284s" totalImagesPullingTime="18.917469533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:58.88313383 +0000 UTC m=+22.649266966" watchObservedRunningTime="2026-07-16 11:08:58.883550284 +0000 UTC m=+22.649683422" Jul 16 11:08:58.904655 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:58.904593 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c5s45" podStartSLOduration=3.9836828 podStartE2EDuration="22.904573007s" totalImagesPullingTime="18.920890207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:58.901524254 +0000 UTC m=+22.667657391" watchObservedRunningTime="2026-07-16 11:08:58.904573007 +0000 UTC m=+22.670706145" Jul 16 11:08:59.797390 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.797132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:08:59.798152 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:08:59.797440 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:08:59.870724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.870685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szwbp" event={"ID":"444e123d-6ada-4f32-8754-d805cb14097a","Type":"ContainerStarted","Data":"f9f6d5db5bc6a8893b3d59d4bf53419661e06dff8fe7040cabe91fa1033bf720"} Jul 16 11:08:59.872612 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.872578 2580 generic.go:356] "Generic (PLEG): container finished" podID="04e9f6389d0c6d59e071fd815e191794" containerID="a9931b9c13dc55669eb6bb982ae4db2d4fd750c55af0bf5f160802b9c2e53b1d" exitCode=0 Jul 16 11:08:59.872729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.872634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" event={"ID":"04e9f6389d0c6d59e071fd815e191794","Type":"ContainerDied","Data":"a9931b9c13dc55669eb6bb982ae4db2d4fd750c55af0bf5f160802b9c2e53b1d"} Jul 16 11:08:59.874664 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.874532 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="ea07d0bcb862bf05488d88e91f12ea94bd2e178066f9694985e81f6c38663434" exitCode=0 Jul 16 11:08:59.874664 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.874605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"ea07d0bcb862bf05488d88e91f12ea94bd2e178066f9694985e81f6c38663434"} Jul 16 11:08:59.878356 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.878321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ll285" event={"ID":"292401f4-8e04-4b7d-b3dc-0604062897e4","Type":"ContainerStarted","Data":"c00eeb6beacbc94a1c6b49eaeb187b6f160466337fbce07fb9f4a6235557dac3"} Jul 16 11:08:59.880136 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.880106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d6gc4" event={"ID":"78778387-11b8-464f-8f4f-b791c7e46005","Type":"ContainerStarted","Data":"7edb48cd47a87c7d9f829e1b59b7ad4e2a1f3979f41f4b5fc20d092c59bbc227"} Jul 16 11:08:59.881600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.881577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" event={"ID":"42c6d016-3953-49a4-b2d0-3268fba259dd","Type":"ContainerStarted","Data":"1e53783c952fabec58d646875362e0d37e3ebf1001c2301dae239a6f818ae25b"} Jul 16 11:08:59.883015 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.882980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-js4ck" event={"ID":"269c6854-9df1-4fb2-a189-c2d56701199e","Type":"ContainerStarted","Data":"3a94eb98e15472a8f9ba70f1a6a798338ceffddf631c0617b47e55125ee0f494"} Jul 16 11:08:59.886250 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886227 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:08:59.886687 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886656 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c15c583-3e8a-4ea6-9cf4-45c3702d0517" containerID="201fc8c7314a9cc97e19cc85b5f782bf9e3f05d6e1f3c91003895854628e1776" exitCode=1 Jul 16 11:08:59.886801 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"b964c5ea607e36dcdd4fc0499e01e41752b6ba977b45f5167db93606818dab6a"} Jul 16 11:08:59.886860 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"a0d813b3c03024ec1e815fe09d58f6f9b2981250dff4e2afca22a45437839503"} Jul 16 11:08:59.886860 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"bb0600179f3ab54036eb2bd79ef593b850c8119ada0aabbf2e4db10532269c71"} Jul 16 11:08:59.886965 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"b0adf5859b997d155d1bcc8e29d862d63b611059838d2689f8acd063c41a184f"} Jul 16 11:08:59.886965 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.886950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerDied","Data":"201fc8c7314a9cc97e19cc85b5f782bf9e3f05d6e1f3c91003895854628e1776"} Jul 16 11:08:59.887066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.887002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"4c47338cfd1f268306b41d06d0cc524170edd12f8ba83d60f997068ec413cb3f"} Jul 16 11:08:59.887413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.887349 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-szwbp" podStartSLOduration=13.03438992 podStartE2EDuration="23.887331283s" totalImagesPullingTime="10.852941363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:59.886967621 +0000 UTC m=+23.653100758" watchObservedRunningTime="2026-07-16 11:08:59.887331283 +0000 UTC m=+23.653464421" Jul 16 11:08:59.905294 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.905232 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-js4ck" podStartSLOduration=4.992577721 podStartE2EDuration="23.905209073s" totalImagesPullingTime="18.912631352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:59.904796929 +0000 UTC m=+23.670930067" watchObservedRunningTime="2026-07-16 11:08:59.905209073 +0000 UTC m=+23.671342212" Jul 16 11:08:59.961060 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.961016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ll285" podStartSLOduration=5.129954028 podStartE2EDuration="23.961001057s" totalImagesPullingTime="18.831047029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:59.960738583 +0000 UTC m=+23.726871720" watchObservedRunningTime="2026-07-16 11:08:59.961001057 +0000 UTC m=+23.727134222" Jul 16 11:08:59.978252 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:08:59.978210 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d6gc4" podStartSLOduration=5.145430678 podStartE2EDuration="23.978194683s" totalImagesPullingTime="18.832764005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:08:59.978057172 +0000 UTC m=+23.744190311" watchObservedRunningTime="2026-07-16 11:08:59.978194683 +0000 UTC m=+23.744327820" Jul 16 11:09:00.470365 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.470337 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 11:09:00.682815 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.682649 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T11:09:00.47035714Z","UUID":"6a413e83-9615-4435-964a-88f033e76a6f","Handler":null,"Name":"","Endpoint":""} Jul 16 11:09:00.684332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.684317 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 11:09:00.684444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.684352 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 11:09:00.796774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.796747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:00.796959 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.796804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:00.796959 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:00.796871 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:00.797033 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:00.796995 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:00.890200 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.890164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" event={"ID":"42c6d016-3953-49a4-b2d0-3268fba259dd","Type":"ContainerStarted","Data":"a54ababfd691841d0795516932fce91d40c632b39fad912c8655145b8696ef78"} Jul 16 11:09:00.892157 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:00.892110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" event={"ID":"04e9f6389d0c6d59e071fd815e191794","Type":"ContainerStarted","Data":"1958cf649d03d3c59b84f0d7cd0f167d752486f839199c9443f480eefa795515"} Jul 16 11:09:01.796749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:01.796713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:01.796964 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:01.796863 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:01.898887 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:01.898848 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:09:01.899393 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:01.899282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"8063502b55f3e95f0700185c3b0aef8bdf1f430e92ad4f375e124b19debd65be"} Jul 16 11:09:02.796694 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:02.796462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:02.796916 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:02.796794 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:02.796916 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:02.796548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:02.796916 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:02.796897 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:03.477295 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.476954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:09:03.478034 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.477786 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:09:03.494121 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.494073 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-16.ec2.internal" podStartSLOduration=26.494056489 podStartE2EDuration="26.494056489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:00.909926581 +0000 UTC m=+24.676059717" watchObservedRunningTime="2026-07-16 11:09:03.494056489 +0000 UTC m=+27.260189625" Jul 16 11:09:03.797282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.797247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:03.797491 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:03.797400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:03.907100 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.907049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" event={"ID":"42c6d016-3953-49a4-b2d0-3268fba259dd","Type":"ContainerStarted","Data":"4bb7ff7fd71f32a4e9a70ceaddbd4c7fbbaff0f189c2eaef6ffcb0786dbde425"} Jul 16 11:09:03.907398 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.907359 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:09:03.907791 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.907774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szwbp" Jul 16 11:09:03.924414 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:03.924341 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4vld" podStartSLOduration=5.113238756 podStartE2EDuration="27.924325972s" totalImagesPullingTime="22.811087216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:03.924067889 +0000 UTC m=+27.690201028" watchObservedRunningTime="2026-07-16 11:09:03.924325972 +0000 UTC m=+27.690459112" Jul 16 11:09:04.796771 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:04.796729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:04.797283 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:04.796783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:04.797283 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:04.796850 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:04.797283 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:04.796947 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:05.797554 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.797357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:05.797926 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:05.797641 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:05.914940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.914869 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:09:05.915249 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.915228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"d9cf8b6fcb0a31415c7b2ccb5484ecfc6efb6038d0f659bbedbd65bf2a132f2e"} Jul 16 11:09:05.915521 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.915501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:05.915735 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.915721 2580 scope.go:117] "RemoveContainer" containerID="201fc8c7314a9cc97e19cc85b5f782bf9e3f05d6e1f3c91003895854628e1776" Jul 16 11:09:05.917161 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.917133 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="526722300e680ffaa0dc80c9ca1e3cbe1efeb3e7d13a6b154c55c7a74569430a" exitCode=0 Jul 16 11:09:05.917281 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.917217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"526722300e680ffaa0dc80c9ca1e3cbe1efeb3e7d13a6b154c55c7a74569430a"} Jul 16 11:09:05.932477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:05.932455 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:06.798463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.798426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:06.799005 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.798477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:06.799005 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:06.798547 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:06.799005 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:06.798609 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:06.924411 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.924368 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:09:06.924797 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.924761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" event={"ID":"1c15c583-3e8a-4ea6-9cf4-45c3702d0517","Type":"ContainerStarted","Data":"a701a9811368930f800f3048e8d2fcfc7c5e49293b1bc2613a768714932b5aa9"} Jul 16 11:09:06.925124 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.925100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:06.925287 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.925248 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:06.940489 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.940461 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:06.953580 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:06.953527 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" podStartSLOduration=11.612624168 podStartE2EDuration="30.953510634s" totalImagesPullingTime="19.340886466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:06.951881254 +0000 UTC m=+30.718014391" watchObservedRunningTime="2026-07-16 11:09:06.953510634 +0000 UTC m=+30.719643771" Jul 16 11:09:07.390624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.390426 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhbv2"] Jul 16 11:09:07.390872 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.390711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:07.390872 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:07.390806 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:07.393620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.393594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkqxh"] Jul 16 11:09:07.393762 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.393722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:07.393861 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:07.393841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:07.394152 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.394131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkddl"] Jul 16 11:09:07.394231 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.394220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:07.394325 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:07.394307 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:07.929232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.929198 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="8eae8be22e42a7041323f67bf8e12ed3d2e0a7d6622f44f232c2def81e94248f" exitCode=0 Jul 16 11:09:07.929634 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:07.929283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"8eae8be22e42a7041323f67bf8e12ed3d2e0a7d6622f44f232c2def81e94248f"} Jul 16 11:09:08.798109 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:08.796957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:08.798189 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:08.798145 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:08.798586 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:08.798251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:08.798586 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:08.798424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:08.798765 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:08.798718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:08.799250 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:08.799029 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:08.933766 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:08.933728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerStarted","Data":"ba395f0ea4d9e8e680d58f99177bf5388629e1a9932af86b751fcdaf9d16c5d4"} Jul 16 11:09:09.939169 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:09.939137 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="ba395f0ea4d9e8e680d58f99177bf5388629e1a9932af86b751fcdaf9d16c5d4" exitCode=0 Jul 16 11:09:09.939595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:09.939199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"ba395f0ea4d9e8e680d58f99177bf5388629e1a9932af86b751fcdaf9d16c5d4"} Jul 16 11:09:10.368444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.368412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:10.368647 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.368563 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:09:10.368647 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.368627 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs podName:48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:42.368607732 +0000 UTC m=+66.134740869 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs") pod "network-metrics-daemon-fkqxh" (UID: "48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 11:09:10.469046 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.469007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:10.469236 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.469180 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 11:09:10.469295 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.469259 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret podName:ee647ad6-cd55-4717-b5ac-618f94505d75 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:42.469237297 +0000 UTC m=+66.235370419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret") pod "global-pull-secret-syncer-jkddl" (UID: "ee647ad6-cd55-4717-b5ac-618f94505d75") : object "kube-system"/"original-pull-secret" not registered Jul 16 11:09:10.569361 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.569329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:10.569548 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.569527 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 11:09:10.569624 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.569559 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 11:09:10.569624 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.569573 2580 projected.go:196] Error preparing data for projected volume kube-api-access-jh2fk for pod openshift-network-diagnostics/network-check-target-bhbv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:09:10.569718 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.569644 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk podName:0be7b124-9cc2-4b47-9cfb-683ce29ddba5 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:42.569619055 +0000 UTC m=+66.335752184 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jh2fk" (UniqueName: "kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk") pod "network-check-target-bhbv2" (UID: "0be7b124-9cc2-4b47-9cfb-683ce29ddba5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 11:09:10.797174 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.797083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:10.797174 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.797116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:10.797394 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.797212 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkddl" podUID="ee647ad6-cd55-4717-b5ac-618f94505d75" Jul 16 11:09:10.797394 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:10.797276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:10.797394 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.797274 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkqxh" podUID="48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4" Jul 16 11:09:10.797394 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:10.797356 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhbv2" podUID="0be7b124-9cc2-4b47-9cfb-683ce29ddba5" Jul 16 11:09:12.486850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.486824 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-16.ec2.internal" event="NodeReady" Jul 16 11:09:12.487364 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.486943 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 11:09:12.536861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.536777 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wr645"] Jul 16 11:09:12.539799 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.539769 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.539946 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.539862 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lgjd4"] Jul 16 11:09:12.541918 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.541893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.543356 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.543337 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 11:09:12.543477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.543371 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnq66\"" Jul 16 11:09:12.543648 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.543626 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 11:09:12.544302 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.544284 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 11:09:12.544455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.544440 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 11:09:12.544578 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.544564 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mgj7r\"" Jul 16 11:09:12.544850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.544829 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 11:09:12.553955 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.553934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wr645"] Jul 16 11:09:12.556451 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.556428 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgjd4"] Jul 16 11:09:12.683471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2j8\" (UniqueName: \"kubernetes.io/projected/048fad95-c706-4a41-9053-6f909977047a-kube-api-access-pq2j8\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.683471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld75q\" (UniqueName: \"kubernetes.io/projected/edac3d95-d82e-45ae-8063-fe14ecc387c5-kube-api-access-ld75q\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.683701 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048fad95-c706-4a41-9053-6f909977047a-tmp-dir\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.683701 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/048fad95-c706-4a41-9053-6f909977047a-config-volume\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.683701 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.683816 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.683698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edac3d95-d82e-45ae-8063-fe14ecc387c5-cert\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.784569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.784569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edac3d95-d82e-45ae-8063-fe14ecc387c5-cert\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2j8\" (UniqueName: \"kubernetes.io/projected/048fad95-c706-4a41-9053-6f909977047a-kube-api-access-pq2j8\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld75q\" (UniqueName: \"kubernetes.io/projected/edac3d95-d82e-45ae-8063-fe14ecc387c5-kube-api-access-ld75q\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:12.784611 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048fad95-c706-4a41-9053-6f909977047a-tmp-dir\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:12.784706 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls podName:048fad95-c706-4a41-9053-6f909977047a nodeName:}" failed. No retries permitted until 2026-07-16 11:09:13.284673834 +0000 UTC m=+37.050806966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls") pod "dns-default-wr645" (UID: "048fad95-c706-4a41-9053-6f909977047a") : secret "dns-default-metrics-tls" not found Jul 16 11:09:12.784810 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.784747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/048fad95-c706-4a41-9053-6f909977047a-config-volume\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.785124 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.785092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048fad95-c706-4a41-9053-6f909977047a-tmp-dir\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.785394 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.785356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/048fad95-c706-4a41-9053-6f909977047a-config-volume\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.789654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.789628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edac3d95-d82e-45ae-8063-fe14ecc387c5-cert\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.793706 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.793680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld75q\" (UniqueName: \"kubernetes.io/projected/edac3d95-d82e-45ae-8063-fe14ecc387c5-kube-api-access-ld75q\") pod \"ingress-canary-lgjd4\" (UID: \"edac3d95-d82e-45ae-8063-fe14ecc387c5\") " pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.796365 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.796343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:12.796608 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.796575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2j8\" (UniqueName: \"kubernetes.io/projected/048fad95-c706-4a41-9053-6f909977047a-kube-api-access-pq2j8\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:12.796895 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.796875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:12.797089 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.797070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:12.799117 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hjhcs\"" Jul 16 11:09:12.799644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 11:09:12.799739 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799649 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 11:09:12.799739 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799669 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 11:09:12.799850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wdm79\"" Jul 16 11:09:12.799850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.799787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 11:09:12.859834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.859788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgjd4" Jul 16 11:09:12.933325 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.933108 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ms5rw"] Jul 16 11:09:12.935798 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.935717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:12.940062 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.940039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hljx9\"" Jul 16 11:09:12.940339 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.940320 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 11:09:12.940611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.940589 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 11:09:12.940961 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.940826 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 11:09:12.940961 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.940840 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 11:09:12.948142 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:12.948066 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ms5rw"] Jul 16 11:09:13.030885 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.030853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgjd4"] Jul 16 11:09:13.034653 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:13.034564 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedac3d95_d82e_45ae_8063_fe14ecc387c5.slice/crio-5cc86780df66f8ed0905d7ffb0450aba926359a6ec453446ed4f945a91ea46af WatchSource:0}: Error finding container 5cc86780df66f8ed0905d7ffb0450aba926359a6ec453446ed4f945a91ea46af: Status 404 returned error can't find the container with id 5cc86780df66f8ed0905d7ffb0450aba926359a6ec453446ed4f945a91ea46af Jul 16 11:09:13.087340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.087301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmm5\" (UniqueName: \"kubernetes.io/projected/46d57d98-2153-4a95-825e-53650d2e7435-kube-api-access-kwmm5\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.087340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.087347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.087602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.087479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.087602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.087509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d57d98-2153-4a95-825e-53650d2e7435-data-volume\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.087602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.087540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d57d98-2153-4a95-825e-53650d2e7435-crio-socket\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188228 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d57d98-2153-4a95-825e-53650d2e7435-data-volume\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188228 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d57d98-2153-4a95-825e-53650d2e7435-crio-socket\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmm5\" (UniqueName: \"kubernetes.io/projected/46d57d98-2153-4a95-825e-53650d2e7435-kube-api-access-kwmm5\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188477 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.188445 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 11:09:13.188653 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.188513 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:13.688499175 +0000 UTC m=+37.454632294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : configmap "kube-rbac-proxy" not found Jul 16 11:09:13.188653 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.188520 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:13.188653 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46d57d98-2153-4a95-825e-53650d2e7435-crio-socket\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.188653 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.188579 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:13.688565718 +0000 UTC m=+37.454698833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:13.188653 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.188595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46d57d98-2153-4a95-825e-53650d2e7435-data-volume\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.207336 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.207312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmm5\" (UniqueName: \"kubernetes.io/projected/46d57d98-2153-4a95-825e-53650d2e7435-kube-api-access-kwmm5\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.288857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.288768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:13.288978 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.288917 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 11:09:13.288978 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.288976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls podName:048fad95-c706-4a41-9053-6f909977047a nodeName:}" failed. No retries permitted until 2026-07-16 11:09:14.288961859 +0000 UTC m=+38.055094973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls") pod "dns-default-wr645" (UID: "048fad95-c706-4a41-9053-6f909977047a") : secret "dns-default-metrics-tls" not found Jul 16 11:09:13.691929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.691898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.692418 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.691962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:13.692418 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.692054 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 11:09:13.692418 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.692080 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:13.692418 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.692136 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:14.692120043 +0000 UTC m=+38.458253162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:13.692418 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:13.692153 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:14.692146487 +0000 UTC m=+38.458279601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : configmap "kube-rbac-proxy" not found Jul 16 11:09:13.953334 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.953255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgjd4" event={"ID":"edac3d95-d82e-45ae-8063-fe14ecc387c5","Type":"ContainerStarted","Data":"5cc86780df66f8ed0905d7ffb0450aba926359a6ec453446ed4f945a91ea46af"} Jul 16 11:09:13.956243 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.956212 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="c77569997efa41b9748f1a354afae21463cd9393c08d3c649f8a2248fa756cca" exitCode=0 Jul 16 11:09:13.956465 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:13.956256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"c77569997efa41b9748f1a354afae21463cd9393c08d3c649f8a2248fa756cca"} Jul 16 11:09:14.297864 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:14.297779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:14.298014 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.297936 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 11:09:14.298014 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.297995 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls podName:048fad95-c706-4a41-9053-6f909977047a nodeName:}" failed. No retries permitted until 2026-07-16 11:09:16.297979184 +0000 UTC m=+40.064112303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls") pod "dns-default-wr645" (UID: "048fad95-c706-4a41-9053-6f909977047a") : secret "dns-default-metrics-tls" not found Jul 16 11:09:14.701359 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:14.701318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:14.701815 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:14.701428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:14.701815 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.701453 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 11:09:14.701815 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.701521 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:16.701500435 +0000 UTC m=+40.467633566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : configmap "kube-rbac-proxy" not found Jul 16 11:09:14.701815 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.701599 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:14.701815 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:14.701675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:16.701656035 +0000 UTC m=+40.467789155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:14.962749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:14.962520 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded" containerID="0a7d70a5abbacc85d6cc8a2f1517d142535515d9a333b664e172c372b4b5c3a4" exitCode=0 Jul 16 11:09:14.962749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:14.962585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerDied","Data":"0a7d70a5abbacc85d6cc8a2f1517d142535515d9a333b664e172c372b4b5c3a4"} Jul 16 11:09:15.899962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:15.899925 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d6gc4_78778387-11b8-464f-8f4f-b791c7e46005/dns-node-resolver/0.log" Jul 16 11:09:15.969693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:15.969659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8npjw" event={"ID":"4bfb2f7d-dfe9-4933-8d6a-ee97d8c7aded","Type":"ContainerStarted","Data":"0cc79b4f3a540e88b6d5ba639c80587dcc444afea291b15ef3e6e2d62e06e77c"} Jul 16 11:09:15.994441 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:15.994367 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8npjw" podStartSLOduration=9.500155574 podStartE2EDuration="39.994347406s" totalImagesPullingTime="30.494191832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:15.991782889 +0000 UTC m=+39.757916050" watchObservedRunningTime="2026-07-16 11:09:15.994347406 +0000 UTC m=+39.760480555" Jul 16 11:09:16.138907 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.138873 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpzrd"] Jul 16 11:09:16.143410 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.143395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.145641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.145595 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 11:09:16.145641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.145614 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 11:09:16.145641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.145598 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 11:09:16.145940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.145650 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fwzgz\"" Jul 16 11:09:16.145940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.145650 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 11:09:16.148896 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.148877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpzrd"] Jul 16 11:09:16.214791 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.214726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjxxr\" (UniqueName: \"kubernetes.io/projected/9270cabf-a73e-426b-962a-cbdad3cda286-kube-api-access-bjxxr\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.214970 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.214830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9270cabf-a73e-426b-962a-cbdad3cda286-signing-key\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.214970 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.214884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9270cabf-a73e-426b-962a-cbdad3cda286-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.316302 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.316275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9270cabf-a73e-426b-962a-cbdad3cda286-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.316486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.316317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:16.316486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.316430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjxxr\" (UniqueName: \"kubernetes.io/projected/9270cabf-a73e-426b-962a-cbdad3cda286-kube-api-access-bjxxr\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.316486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.316471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9270cabf-a73e-426b-962a-cbdad3cda286-signing-key\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.316486 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.316476 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 11:09:16.316691 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.316551 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls podName:048fad95-c706-4a41-9053-6f909977047a nodeName:}" failed. No retries permitted until 2026-07-16 11:09:20.316531053 +0000 UTC m=+44.082664186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls") pod "dns-default-wr645" (UID: "048fad95-c706-4a41-9053-6f909977047a") : secret "dns-default-metrics-tls" not found Jul 16 11:09:16.317016 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.316996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9270cabf-a73e-426b-962a-cbdad3cda286-signing-cabundle\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.319602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.319582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9270cabf-a73e-426b-962a-cbdad3cda286-signing-key\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.325003 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.324982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjxxr\" (UniqueName: \"kubernetes.io/projected/9270cabf-a73e-426b-962a-cbdad3cda286-kube-api-access-bjxxr\") pod \"service-ca-79f68b9f4f-bpzrd\" (UID: \"9270cabf-a73e-426b-962a-cbdad3cda286\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.455970 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.455837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" Jul 16 11:09:16.603419 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.603368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bpzrd"] Jul 16 11:09:16.607304 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:16.607276 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9270cabf_a73e_426b_962a_cbdad3cda286.slice/crio-4381a0012f660b8db1f657d1d1b78467ec2330dada3d9adcf8d494c444415739 WatchSource:0}: Error finding container 4381a0012f660b8db1f657d1d1b78467ec2330dada3d9adcf8d494c444415739: Status 404 returned error can't find the container with id 4381a0012f660b8db1f657d1d1b78467ec2330dada3d9adcf8d494c444415739 Jul 16 11:09:16.700126 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.700098 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-js4ck_269c6854-9df1-4fb2-a189-c2d56701199e/node-ca/0.log" Jul 16 11:09:16.720569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.720490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:16.720677 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.720592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:16.720677 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.720630 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 11:09:16.720774 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.720699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:20.720680935 +0000 UTC m=+44.486814050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : configmap "kube-rbac-proxy" not found Jul 16 11:09:16.720956 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.720784 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:16.720956 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:16.720868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:20.720849902 +0000 UTC m=+44.486983036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:16.973611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.973530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgjd4" event={"ID":"edac3d95-d82e-45ae-8063-fe14ecc387c5","Type":"ContainerStarted","Data":"2b0213fb2a1e92acd523a071ec06ca26e2666ea4bf33ffd951a41e1c3ea93104"} Jul 16 11:09:16.974689 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.974662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" event={"ID":"9270cabf-a73e-426b-962a-cbdad3cda286","Type":"ContainerStarted","Data":"4381a0012f660b8db1f657d1d1b78467ec2330dada3d9adcf8d494c444415739"} Jul 16 11:09:16.991438 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:16.991361 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lgjd4" podStartSLOduration=1.805146949 podStartE2EDuration="4.991341748s" totalImagesPullingTime="3.186194799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:16.991105361 +0000 UTC m=+40.757238502" watchObservedRunningTime="2026-07-16 11:09:16.991341748 +0000 UTC m=+40.757474920" Jul 16 11:09:17.500455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:17.500424 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgjd4_edac3d95-d82e-45ae-8063-fe14ecc387c5/serve-healthcheck-canary/0.log" Jul 16 11:09:19.982639 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:19.982600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" event={"ID":"9270cabf-a73e-426b-962a-cbdad3cda286","Type":"ContainerStarted","Data":"505ec564832191068c35b57ef294bbb43188ce6810406a0316269874bd423ec5"} Jul 16 11:09:20.350875 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:20.350778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:20.351012 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.350932 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 11:09:20.351012 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.351007 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls podName:048fad95-c706-4a41-9053-6f909977047a nodeName:}" failed. No retries permitted until 2026-07-16 11:09:28.350989303 +0000 UTC m=+52.117122422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls") pod "dns-default-wr645" (UID: "048fad95-c706-4a41-9053-6f909977047a") : secret "dns-default-metrics-tls" not found Jul 16 11:09:20.753217 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:20.753172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:20.753421 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:20.753246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:20.753421 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.753314 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 11:09:20.753421 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.753366 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:20.753421 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.753414 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:28.753392274 +0000 UTC m=+52.519525404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : configmap "kube-rbac-proxy" not found Jul 16 11:09:20.753576 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:20.753434 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:28.753425065 +0000 UTC m=+52.519558195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:28.414280 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.414239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:28.416663 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.416635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/048fad95-c706-4a41-9053-6f909977047a-metrics-tls\") pod \"dns-default-wr645\" (UID: \"048fad95-c706-4a41-9053-6f909977047a\") " pod="openshift-dns/dns-default-wr645" Jul 16 11:09:28.452926 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.452885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wr645" Jul 16 11:09:28.605856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.605801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-bpzrd" podStartSLOduration=10.191160326 podStartE2EDuration="12.605783651s" totalImagesPullingTime="2.414623325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:19.99918833 +0000 UTC m=+43.765321462" watchObservedRunningTime="2026-07-16 11:09:28.605783651 +0000 UTC m=+52.371916787" Jul 16 11:09:28.606086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.606071 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wr645"] Jul 16 11:09:28.609478 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:28.609447 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048fad95_c706_4a41_9053_6f909977047a.slice/crio-69e1ff8882a7928293df927efa0aa333fed761921344c6681acd571778404dcc WatchSource:0}: Error finding container 69e1ff8882a7928293df927efa0aa333fed761921344c6681acd571778404dcc: Status 404 returned error can't find the container with id 69e1ff8882a7928293df927efa0aa333fed761921344c6681acd571778404dcc Jul 16 11:09:28.818322 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.818235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:28.818322 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.818311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:28.818517 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:28.818421 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 11:09:28.818517 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:28.818489 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls podName:46d57d98-2153-4a95-825e-53650d2e7435 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:44.818472397 +0000 UTC m=+68.584605516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms5rw" (UID: "46d57d98-2153-4a95-825e-53650d2e7435") : secret "insights-runtime-extractor-tls" not found Jul 16 11:09:28.818821 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:28.818804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46d57d98-2153-4a95-825e-53650d2e7435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:29.007449 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:29.007415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr645" event={"ID":"048fad95-c706-4a41-9053-6f909977047a","Type":"ContainerStarted","Data":"69e1ff8882a7928293df927efa0aa333fed761921344c6681acd571778404dcc"} Jul 16 11:09:31.014808 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:31.014714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr645" event={"ID":"048fad95-c706-4a41-9053-6f909977047a","Type":"ContainerStarted","Data":"4c5aca1cc9c4689e23efcda59f09479478af6411b5543216fa617b47f144b599"} Jul 16 11:09:31.014808 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:31.014794 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr645" event={"ID":"048fad95-c706-4a41-9053-6f909977047a","Type":"ContainerStarted","Data":"9790b075b5af7c9e0966af13fc9edf31ea9050971345db0253683987d4d2e263"} Jul 16 11:09:31.015201 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:31.014859 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wr645" Jul 16 11:09:31.033556 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:31.033513 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wr645" podStartSLOduration=17.029780915 podStartE2EDuration="19.03349894s" totalImagesPullingTime="2.003718025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:31.032508213 +0000 UTC m=+54.798641347" watchObservedRunningTime="2026-07-16 11:09:31.03349894 +0000 UTC m=+54.799632077" Jul 16 11:09:38.943966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:38.943932 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrmqc" Jul 16 11:09:39.108770 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.108735 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-c6vd6"] Jul 16 11:09:39.111563 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.111546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:09:39.114772 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.114467 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 11:09:39.114772 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.114514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 11:09:39.114772 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.114514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lsmjs\"" Jul 16 11:09:39.123148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.123123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-c6vd6"] Jul 16 11:09:39.189242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.189203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwspr\" (UniqueName: \"kubernetes.io/projected/b7cd9313-0cd3-42e5-b793-5adeddf848d0-kube-api-access-kwspr\") pod \"downloads-68c9cbb497-c6vd6\" (UID: \"b7cd9313-0cd3-42e5-b793-5adeddf848d0\") " pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:09:39.216714 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.216642 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c7989975f-hxnhd"] Jul 16 11:09:39.218476 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.218453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.221143 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.221105 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 11:09:39.221309 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.221287 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 11:09:39.221722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.221703 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bq7p2\"" Jul 16 11:09:39.221821 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.221736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 11:09:39.226972 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.226954 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 11:09:39.237249 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.237222 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c7989975f-hxnhd"] Jul 16 11:09:39.290470 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-installation-pull-secrets\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290652 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-image-registry-private-configuration\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290652 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwspr\" (UniqueName: \"kubernetes.io/projected/b7cd9313-0cd3-42e5-b793-5adeddf848d0-kube-api-access-kwspr\") pod \"downloads-68c9cbb497-c6vd6\" (UID: \"b7cd9313-0cd3-42e5-b793-5adeddf848d0\") " pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:09:39.290652 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-registry-tls\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290652 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-registry-certificates\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19799a0e-030d-4e24-961a-26122343409d-ca-trust-extracted\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblg9\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-kube-api-access-lblg9\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-bound-sa-token\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.290838 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.290805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-trusted-ca\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.299277 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.299252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwspr\" (UniqueName: \"kubernetes.io/projected/b7cd9313-0cd3-42e5-b793-5adeddf848d0-kube-api-access-kwspr\") pod \"downloads-68c9cbb497-c6vd6\" (UID: \"b7cd9313-0cd3-42e5-b793-5adeddf848d0\") " pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:09:39.391120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19799a0e-030d-4e24-961a-26122343409d-ca-trust-extracted\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391120 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblg9\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-kube-api-access-lblg9\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391424 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-bound-sa-token\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391424 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-trusted-ca\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391424 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-installation-pull-secrets\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391590 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-image-registry-private-configuration\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391590 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19799a0e-030d-4e24-961a-26122343409d-ca-trust-extracted\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391590 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-registry-tls\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.391590 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.391583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-registry-certificates\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.392263 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.392229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-registry-certificates\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.392469 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.392446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19799a0e-030d-4e24-961a-26122343409d-trusted-ca\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.393880 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.393857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-registry-tls\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.393971 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.393894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-installation-pull-secrets\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.393971 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.393905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19799a0e-030d-4e24-961a-26122343409d-image-registry-private-configuration\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.400168 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.400129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-bound-sa-token\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.400815 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.400792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblg9\" (UniqueName: \"kubernetes.io/projected/19799a0e-030d-4e24-961a-26122343409d-kube-api-access-lblg9\") pod \"image-registry-5c7989975f-hxnhd\" (UID: \"19799a0e-030d-4e24-961a-26122343409d\") " pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.423738 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.423706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:09:39.529286 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.529253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:39.548366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.548333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-c6vd6"] Jul 16 11:09:39.551980 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:39.551955 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cd9313_0cd3_42e5_b793_5adeddf848d0.slice/crio-6cb4359b2a27f1d98eb09689a1db89481e1a68cf2bc29dfa1fdb8a2278d694aa WatchSource:0}: Error finding container 6cb4359b2a27f1d98eb09689a1db89481e1a68cf2bc29dfa1fdb8a2278d694aa: Status 404 returned error can't find the container with id 6cb4359b2a27f1d98eb09689a1db89481e1a68cf2bc29dfa1fdb8a2278d694aa Jul 16 11:09:39.658174 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:39.658144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c7989975f-hxnhd"] Jul 16 11:09:39.661008 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:39.660983 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19799a0e_030d_4e24_961a_26122343409d.slice/crio-f9059fc32aad874646fc94cdce3a4cf0ec0d21b60285ccd5c93abd80c4e91e6d WatchSource:0}: Error finding container f9059fc32aad874646fc94cdce3a4cf0ec0d21b60285ccd5c93abd80c4e91e6d: Status 404 returned error can't find the container with id f9059fc32aad874646fc94cdce3a4cf0ec0d21b60285ccd5c93abd80c4e91e6d Jul 16 11:09:40.041188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:40.041108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-c6vd6" event={"ID":"b7cd9313-0cd3-42e5-b793-5adeddf848d0","Type":"ContainerStarted","Data":"6cb4359b2a27f1d98eb09689a1db89481e1a68cf2bc29dfa1fdb8a2278d694aa"} Jul 16 11:09:40.043198 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:40.043166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" event={"ID":"19799a0e-030d-4e24-961a-26122343409d","Type":"ContainerStarted","Data":"b806f1144d6d2c81762f4803fbd0a023bf40c58645aeb5d969461fc048fff170"} Jul 16 11:09:40.043362 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:40.043251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" event={"ID":"19799a0e-030d-4e24-961a-26122343409d","Type":"ContainerStarted","Data":"f9059fc32aad874646fc94cdce3a4cf0ec0d21b60285ccd5c93abd80c4e91e6d"} Jul 16 11:09:40.043362 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:40.043329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:09:40.064119 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:40.064035 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" podStartSLOduration=1.064014617 podStartE2EDuration="1.064014617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:40.063004565 +0000 UTC m=+63.829137701" watchObservedRunningTime="2026-07-16 11:09:40.064014617 +0000 UTC m=+63.830147752" Jul 16 11:09:41.020065 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.020031 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wr645" Jul 16 11:09:41.020804 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.020779 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv"] Jul 16 11:09:41.026792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.026763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:41.028735 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.028686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-svplx\"" Jul 16 11:09:41.028735 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.028723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 11:09:41.032993 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.032939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv"] Jul 16 11:09:41.105734 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.104670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ed9f36b-ee8c-4c10-a052-b5cea7a930c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-k2kmv\" (UID: \"5ed9f36b-ee8c-4c10-a052-b5cea7a930c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:41.205476 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.205435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ed9f36b-ee8c-4c10-a052-b5cea7a930c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-k2kmv\" (UID: \"5ed9f36b-ee8c-4c10-a052-b5cea7a930c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:41.208067 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.208038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ed9f36b-ee8c-4c10-a052-b5cea7a930c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-k2kmv\" (UID: \"5ed9f36b-ee8c-4c10-a052-b5cea7a930c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:41.339731 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.339644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:41.483423 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:41.483387 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv"] Jul 16 11:09:41.488016 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:41.487988 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed9f36b_ee8c_4c10_a052_b5cea7a930c4.slice/crio-207cf3cdef2647371b0ed1657d6041c1d347cb28464fbc0c7c1026f4a25eedd6 WatchSource:0}: Error finding container 207cf3cdef2647371b0ed1657d6041c1d347cb28464fbc0c7c1026f4a25eedd6: Status 404 returned error can't find the container with id 207cf3cdef2647371b0ed1657d6041c1d347cb28464fbc0c7c1026f4a25eedd6 Jul 16 11:09:42.053433 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.053393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" event={"ID":"5ed9f36b-ee8c-4c10-a052-b5cea7a930c4","Type":"ContainerStarted","Data":"207cf3cdef2647371b0ed1657d6041c1d347cb28464fbc0c7c1026f4a25eedd6"} Jul 16 11:09:42.415896 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.415853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:42.418581 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.418558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 11:09:42.428762 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.428733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4-metrics-certs\") pod \"network-metrics-daemon-fkqxh\" (UID: \"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4\") " pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:42.517042 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.517006 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hjhcs\"" Jul 16 11:09:42.517228 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.517131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:42.519059 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.519037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 11:09:42.525871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.525846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkqxh" Jul 16 11:09:42.530278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.530244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee647ad6-cd55-4717-b5ac-618f94505d75-original-pull-secret\") pod \"global-pull-secret-syncer-jkddl\" (UID: \"ee647ad6-cd55-4717-b5ac-618f94505d75\") " pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:42.618053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.618014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:42.620927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.620897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 11:09:42.631081 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.631040 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 11:09:42.642540 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.642501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2fk\" (UniqueName: \"kubernetes.io/projected/0be7b124-9cc2-4b47-9cfb-683ce29ddba5-kube-api-access-jh2fk\") pod \"network-check-target-bhbv2\" (UID: \"0be7b124-9cc2-4b47-9cfb-683ce29ddba5\") " pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:42.819501 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.819467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkqxh"] Jul 16 11:09:42.823016 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:42.822984 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f10932_cf6c_4dd1_a2c6_c0ccd54a4fa4.slice/crio-96af7650dd0d958472afbe33bea05903d7dca58d2ef0c0c1f2ff4a98688d4648 WatchSource:0}: Error finding container 96af7650dd0d958472afbe33bea05903d7dca58d2ef0c0c1f2ff4a98688d4648: Status 404 returned error can't find the container with id 96af7650dd0d958472afbe33bea05903d7dca58d2ef0c0c1f2ff4a98688d4648 Jul 16 11:09:42.823454 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.823435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wdm79\"" Jul 16 11:09:42.826863 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.826838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkddl" Jul 16 11:09:42.831943 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.831922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:42.971118 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.971033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkddl"] Jul 16 11:09:42.975448 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:42.975416 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee647ad6_cd55_4717_b5ac_618f94505d75.slice/crio-caef37ad882e55133eae3efaa80a6cd29a6a57167c10275c810e17af8e8a0cf5 WatchSource:0}: Error finding container caef37ad882e55133eae3efaa80a6cd29a6a57167c10275c810e17af8e8a0cf5: Status 404 returned error can't find the container with id caef37ad882e55133eae3efaa80a6cd29a6a57167c10275c810e17af8e8a0cf5 Jul 16 11:09:42.990878 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:42.990849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhbv2"] Jul 16 11:09:42.993954 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:42.993928 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be7b124_9cc2_4b47_9cfb_683ce29ddba5.slice/crio-1af09a586027453c38e0c88ed49ad1428b7211838684c9ea858793b4b8d05119 WatchSource:0}: Error finding container 1af09a586027453c38e0c88ed49ad1428b7211838684c9ea858793b4b8d05119: Status 404 returned error can't find the container with id 1af09a586027453c38e0c88ed49ad1428b7211838684c9ea858793b4b8d05119 Jul 16 11:09:43.058061 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:43.058027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhbv2" event={"ID":"0be7b124-9cc2-4b47-9cfb-683ce29ddba5","Type":"ContainerStarted","Data":"1af09a586027453c38e0c88ed49ad1428b7211838684c9ea858793b4b8d05119"} Jul 16 11:09:43.059346 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:43.059320 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkqxh" event={"ID":"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4","Type":"ContainerStarted","Data":"96af7650dd0d958472afbe33bea05903d7dca58d2ef0c0c1f2ff4a98688d4648"} Jul 16 11:09:43.060734 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:43.060706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkddl" event={"ID":"ee647ad6-cd55-4717-b5ac-618f94505d75","Type":"ContainerStarted","Data":"caef37ad882e55133eae3efaa80a6cd29a6a57167c10275c810e17af8e8a0cf5"} Jul 16 11:09:43.062331 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:43.062307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" event={"ID":"5ed9f36b-ee8c-4c10-a052-b5cea7a930c4","Type":"ContainerStarted","Data":"03d6681edbfb51e9748f31573a61b7bbeb9c848d4b62ed9eda825f7a68b35337"} Jul 16 11:09:43.081161 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:43.081105 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" podStartSLOduration=0.876379842 podStartE2EDuration="2.081086454s" totalImagesPullingTime="1.204706612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:43.079706168 +0000 UTC m=+66.845839306" watchObservedRunningTime="2026-07-16 11:09:43.081086454 +0000 UTC m=+66.847219594" Jul 16 11:09:44.066841 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.066779 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:44.073649 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.073432 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-k2kmv" Jul 16 11:09:44.101512 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.100527 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:09:44.104079 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.104012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.108565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.107975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 11:09:44.108565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.108092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 11:09:44.108565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.108189 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 11:09:44.108565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.108297 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 11:09:44.108565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.108397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrlxb\"" Jul 16 11:09:44.117487 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.113364 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 11:09:44.124068 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.123599 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:09:44.131451 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhj4\" (UniqueName: \"kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.131572 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.131572 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.131572 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.131572 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.131755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.131597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhj4\" (UniqueName: \"kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.232524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.234162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.233234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.235256 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.235208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.235358 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.235336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.236848 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.236806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.242147 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.242102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.247232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.247201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhj4\" (UniqueName: \"kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4\") pod \"console-84d9849dd8-gqcft\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.425271 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.425046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:44.630867 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.630808 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:09:44.635942 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:44.635919 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd9727_16e2_422c_bfa7_607cc282572a.slice/crio-11a7e4a3319c3c00998a02fba765ac2d7c42ab6b16adb280be0eb70355703a31 WatchSource:0}: Error finding container 11a7e4a3319c3c00998a02fba765ac2d7c42ab6b16adb280be0eb70355703a31: Status 404 returned error can't find the container with id 11a7e4a3319c3c00998a02fba765ac2d7c42ab6b16adb280be0eb70355703a31 Jul 16 11:09:44.836616 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.836581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:44.840128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:44.840085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46d57d98-2153-4a95-825e-53650d2e7435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms5rw\" (UID: \"46d57d98-2153-4a95-825e-53650d2e7435\") " pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:45.055819 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.055531 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hljx9\"" Jul 16 11:09:45.063973 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.063643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ms5rw" Jul 16 11:09:45.076259 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.075510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkqxh" event={"ID":"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4","Type":"ContainerStarted","Data":"78b68420aa05ba3eebcecbffce3411f448acafed7c230c06d41cfda1fd09a53e"} Jul 16 11:09:45.076259 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.075580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkqxh" event={"ID":"48f10932-cf6c-4dd1-a2c6-c0ccd54a4fa4","Type":"ContainerStarted","Data":"ad19424ef7220ff63ac0b31735128bf4db6fc1d025f3674dee2a58560af55562"} Jul 16 11:09:45.080902 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.080838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9849dd8-gqcft" event={"ID":"12cd9727-16e2-422c-bfa7-607cc282572a","Type":"ContainerStarted","Data":"11a7e4a3319c3c00998a02fba765ac2d7c42ab6b16adb280be0eb70355703a31"} Jul 16 11:09:45.089649 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.089618 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9tqkw"] Jul 16 11:09:45.095987 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.095109 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fkqxh" podStartSLOduration=67.718651278 podStartE2EDuration="1m9.095084336s" totalImagesPullingTime="1.376433058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:45.093596202 +0000 UTC m=+68.859729342" watchObservedRunningTime="2026-07-16 11:09:45.095084336 +0000 UTC m=+68.861217473" Jul 16 11:09:45.099621 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.099593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.103548 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.103472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9tqkw"] Jul 16 11:09:45.105055 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.104477 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 11:09:45.105055 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.104549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q2z5m\"" Jul 16 11:09:45.105055 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.104588 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 11:09:45.105055 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.104730 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 11:09:45.105055 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.105007 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 11:09:45.105408 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.105069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 11:09:45.138966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.138503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.138966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.138550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfnd\" (UniqueName: \"kubernetes.io/projected/127e89ac-fe5a-4471-a90e-283f0d80aed9-kube-api-access-mdfnd\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.138966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.138589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.138966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.138641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/127e89ac-fe5a-4471-a90e-283f0d80aed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.239412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/127e89ac-fe5a-4471-a90e-283f0d80aed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.239498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.239529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfnd\" (UniqueName: \"kubernetes.io/projected/127e89ac-fe5a-4471-a90e-283f0d80aed9-kube-api-access-mdfnd\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.239563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:45.240081 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 11:09:45.240471 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:09:45.240238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls podName:127e89ac-fe5a-4471-a90e-283f0d80aed9 nodeName:}" failed. No retries permitted until 2026-07-16 11:09:45.740133856 +0000 UTC m=+69.506266975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls") pod "prometheus-operator-64785b6675-9tqkw" (UID: "127e89ac-fe5a-4471-a90e-283f0d80aed9") : secret "prometheus-operator-tls" not found Jul 16 11:09:45.241307 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.241253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/127e89ac-fe5a-4471-a90e-283f0d80aed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.243638 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.243591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.252307 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.252247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfnd\" (UniqueName: \"kubernetes.io/projected/127e89ac-fe5a-4471-a90e-283f0d80aed9-kube-api-access-mdfnd\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.269481 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.269391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ms5rw"] Jul 16 11:09:45.274009 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:45.273967 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d57d98_2153_4a95_825e_53650d2e7435.slice/crio-5e12f09bd4a3caacd582d7c05f72be9b6a79bbc5f64ae84c62284d047c66367c WatchSource:0}: Error finding container 5e12f09bd4a3caacd582d7c05f72be9b6a79bbc5f64ae84c62284d047c66367c: Status 404 returned error can't find the container with id 5e12f09bd4a3caacd582d7c05f72be9b6a79bbc5f64ae84c62284d047c66367c Jul 16 11:09:45.743793 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.743577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:45.747802 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:45.747771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/127e89ac-fe5a-4471-a90e-283f0d80aed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9tqkw\" (UID: \"127e89ac-fe5a-4471-a90e-283f0d80aed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:46.015825 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:46.015747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" Jul 16 11:09:46.092855 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:46.092735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms5rw" event={"ID":"46d57d98-2153-4a95-825e-53650d2e7435","Type":"ContainerStarted","Data":"5e12f09bd4a3caacd582d7c05f72be9b6a79bbc5f64ae84c62284d047c66367c"} Jul 16 11:09:50.221103 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:50.220217 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9tqkw"] Jul 16 11:09:50.226874 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:09:50.226834 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127e89ac_fe5a_4471_a90e_283f0d80aed9.slice/crio-dd99c02d101d58102cb642d5da9e532ed0a228163cb1d7afd32ea1aab7c516b2 WatchSource:0}: Error finding container dd99c02d101d58102cb642d5da9e532ed0a228163cb1d7afd32ea1aab7c516b2: Status 404 returned error can't find the container with id dd99c02d101d58102cb642d5da9e532ed0a228163cb1d7afd32ea1aab7c516b2 Jul 16 11:09:51.124647 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.124607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhbv2" event={"ID":"0be7b124-9cc2-4b47-9cfb-683ce29ddba5","Type":"ContainerStarted","Data":"02225ee16fd441a67c696daa75de65e0bcd42aec7a6103aeb6e48f34da7c0c18"} Jul 16 11:09:51.125005 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.124963 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:09:51.126895 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.126864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9849dd8-gqcft" event={"ID":"12cd9727-16e2-422c-bfa7-607cc282572a","Type":"ContainerStarted","Data":"0d54a502faf2e251fa5ab7dba1a12a14855545e6b686f0207e8d810134a9df90"} Jul 16 11:09:51.128577 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.128548 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" event={"ID":"127e89ac-fe5a-4471-a90e-283f0d80aed9","Type":"ContainerStarted","Data":"dd99c02d101d58102cb642d5da9e532ed0a228163cb1d7afd32ea1aab7c516b2"} Jul 16 11:09:51.130616 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.130588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkddl" event={"ID":"ee647ad6-cd55-4717-b5ac-618f94505d75","Type":"ContainerStarted","Data":"4b75bba4ca6e60963bd09fd8a406be97ec72792dad0fc41a9c1fd5d351b9b195"} Jul 16 11:09:51.146292 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.146189 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bhbv2" podStartSLOduration=68.080741142 podStartE2EDuration="1m15.146164633s" totalImagesPullingTime="7.065423491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:51.145163417 +0000 UTC m=+74.911296556" watchObservedRunningTime="2026-07-16 11:09:51.146164633 +0000 UTC m=+74.912297772" Jul 16 11:09:51.164167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.164064 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jkddl" podStartSLOduration=68.073496753 podStartE2EDuration="1m15.164041268s" totalImagesPullingTime="7.090544515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:51.162890865 +0000 UTC m=+74.929024008" watchObservedRunningTime="2026-07-16 11:09:51.164041268 +0000 UTC m=+74.930174387" Jul 16 11:09:51.183461 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:51.183399 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84d9849dd8-gqcft" podStartSLOduration=1.761750599 podStartE2EDuration="7.183330867s" totalImagesPullingTime="5.421580268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:09:51.18184661 +0000 UTC m=+74.947979763" watchObservedRunningTime="2026-07-16 11:09:51.183330867 +0000 UTC m=+74.949464022" Jul 16 11:09:54.425755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:54.425700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:54.426258 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:54.425781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:54.431492 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:54.431467 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:09:55.149834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:09:55.149803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:10:01.052497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:01.052388 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c7989975f-hxnhd" Jul 16 11:10:01.110677 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:01.110647 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:10:03.179395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.178888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" event={"ID":"127e89ac-fe5a-4471-a90e-283f0d80aed9","Type":"ContainerStarted","Data":"85cb244aa6cddd2ff5f0deecff1cfb5a770c64f509ff94d94c12a5136ef99e48"} Jul 16 11:10:03.179395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.178958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" event={"ID":"127e89ac-fe5a-4471-a90e-283f0d80aed9","Type":"ContainerStarted","Data":"e55bc6f11fdad2724b6a9f739985ab8a093ad8a4d6fb36787a3dc156ee0e3639"} Jul 16 11:10:03.180943 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.180907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-c6vd6" event={"ID":"b7cd9313-0cd3-42e5-b793-5adeddf848d0","Type":"ContainerStarted","Data":"77361a754aceaef1689a23919c0367ca7505fd72e929b8d131e9783493147ae5"} Jul 16 11:10:03.181112 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.181087 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:10:03.183047 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.182777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms5rw" event={"ID":"46d57d98-2153-4a95-825e-53650d2e7435","Type":"ContainerStarted","Data":"24d9cc735e5fceef13f86090f06abc9f573f7200085979ba3bed739e2bd9fe8d"} Jul 16 11:10:03.199943 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.199917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-c6vd6" Jul 16 11:10:03.205175 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.205128 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-9tqkw" podStartSLOduration=6.108597488 podStartE2EDuration="18.20511476s" totalImagesPullingTime="12.096517272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:03.204847713 +0000 UTC m=+86.970980852" watchObservedRunningTime="2026-07-16 11:10:03.20511476 +0000 UTC m=+86.971247896" Jul 16 11:10:03.239828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:03.239769 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-c6vd6" podStartSLOduration=1.4644755950000001 podStartE2EDuration="24.23974461s" totalImagesPullingTime="22.775269015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:03.238852326 +0000 UTC m=+87.004985502" watchObservedRunningTime="2026-07-16 11:10:03.23974461 +0000 UTC m=+87.005877747" Jul 16 11:10:05.520766 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.519330 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kb6t9"] Jul 16 11:10:05.604395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.603432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.609397 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.606651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 11:10:05.609397 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.606865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 11:10:05.609397 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.607137 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 11:10:05.609397 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.607345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qvwlj\"" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-tls\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-metrics-client-ca\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgn6h\" (UniqueName: \"kubernetes.io/projected/5c5ed361-19c0-4de3-8efa-0ea720588bd4-kube-api-access-jgn6h\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-textfile\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-sys\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-wtmp\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.685399 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.683337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-root\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784116 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-tls\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784116 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784116 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-metrics-client-ca\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgn6h\" (UniqueName: \"kubernetes.io/projected/5c5ed361-19c0-4de3-8efa-0ea720588bd4-kube-api-access-jgn6h\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-textfile\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-sys\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-wtmp\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-root\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-root\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784732 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-wtmp\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784732 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5ed361-19c0-4de3-8efa-0ea720588bd4-sys\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.784813 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.784759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-textfile\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.785229 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.785207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-metrics-client-ca\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.785765 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.785449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.787505 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.787460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-tls\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.787618 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.787526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c5ed361-19c0-4de3-8efa-0ea720588bd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.800938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.800887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgn6h\" (UniqueName: \"kubernetes.io/projected/5c5ed361-19c0-4de3-8efa-0ea720588bd4-kube-api-access-jgn6h\") pod \"node-exporter-kb6t9\" (UID: \"5c5ed361-19c0-4de3-8efa-0ea720588bd4\") " pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:05.945275 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:05.945067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kb6t9" Jul 16 11:10:06.539463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.537583 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:10:06.621189 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.621152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:10:06.621437 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.621340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.627811 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.627457 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 11:10:06.627811 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.627617 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 11:10:06.627811 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.627714 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 11:10:06.628128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.627865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 11:10:06.628128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.627886 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 11:10:06.628128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.628028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 11:10:06.628128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.628083 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 11:10:06.628404 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.628226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 11:10:06.628404 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.628244 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sj7dp\"" Jul 16 11:10:06.628512 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.628412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 11:10:06.695731 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.695696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696004 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.695982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696211 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696211 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696211 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696211 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.696587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.696467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkmr\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkmr\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.797595 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.797589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.799924 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.799893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.801769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.800394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.801769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.801469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.802949 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.802906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.803527 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.803500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.804921 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.804874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.805852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.807449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.807519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.807911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.808324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.809227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.808706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.812481 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.812457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkmr\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr\") pod \"alertmanager-main-0\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:06.940596 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:06.940554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:10:08.147709 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:10:08.147645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5ed361_19c0_4de3_8efa_0ea720588bd4.slice/crio-48867965e55d95ca64cf86ea214b311fd5afd5a008d31afe7bc23370ad456238 WatchSource:0}: Error finding container 48867965e55d95ca64cf86ea214b311fd5afd5a008d31afe7bc23370ad456238: Status 404 returned error can't find the container with id 48867965e55d95ca64cf86ea214b311fd5afd5a008d31afe7bc23370ad456238 Jul 16 11:10:08.209191 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:08.208700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb6t9" event={"ID":"5c5ed361-19c0-4de3-8efa-0ea720588bd4","Type":"ContainerStarted","Data":"48867965e55d95ca64cf86ea214b311fd5afd5a008d31afe7bc23370ad456238"} Jul 16 11:10:08.303288 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:08.302904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:10:09.215324 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:09.215282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms5rw" event={"ID":"46d57d98-2153-4a95-825e-53650d2e7435","Type":"ContainerStarted","Data":"f95eb29179f011305da84cc13a7c731ad3a11807e1968bf7d8bff101630f8945"} Jul 16 11:10:09.217231 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:09.217166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"83c8a0862206f0f1a5ad70caef0563cdabe9ed667624a5ead6885dea2a4f60e8"} Jul 16 11:10:10.288929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.287716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:10.312350 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.312133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:10.312350 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.312286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.323842 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.323807 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 11:10:10.431186 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8f58\" (UniqueName: \"kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.431482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.431436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.532886 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.532266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8f58\" (UniqueName: \"kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.533809 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.533342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.534047 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.534022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.534455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.534348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.534538 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.534468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.536871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.536845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.537344 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.537301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.545636 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.545567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8f58\" (UniqueName: \"kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58\") pod \"console-748557646f-x82t4\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.628655 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.628614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:10.977109 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:10.977072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:10.977974 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:10:10.977940 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25144400_4cfe_4716_9145_c615ac45b250.slice/crio-005883ba807831da5406da117f2056a29b69ac7c6da0b77dfb0f078865829fc3 WatchSource:0}: Error finding container 005883ba807831da5406da117f2056a29b69ac7c6da0b77dfb0f078865829fc3: Status 404 returned error can't find the container with id 005883ba807831da5406da117f2056a29b69ac7c6da0b77dfb0f078865829fc3 Jul 16 11:10:11.230192 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.229517 2580 generic.go:356] "Generic (PLEG): container finished" podID="5c5ed361-19c0-4de3-8efa-0ea720588bd4" containerID="c598b30a7b7514f5d5fd423664f8409d97baf04384032a706bd7be9eadee3216" exitCode=0 Jul 16 11:10:11.230192 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.229618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb6t9" event={"ID":"5c5ed361-19c0-4de3-8efa-0ea720588bd4","Type":"ContainerDied","Data":"c598b30a7b7514f5d5fd423664f8409d97baf04384032a706bd7be9eadee3216"} Jul 16 11:10:11.236366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.235222 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf" exitCode=0 Jul 16 11:10:11.236366 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.236221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf"} Jul 16 11:10:11.244637 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.244609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748557646f-x82t4" event={"ID":"25144400-4cfe-4716-9145-c615ac45b250","Type":"ContainerStarted","Data":"12a2d104bae39451d18a760a35556580ee80de0ea8719de5e00e2fb2d8f14cff"} Jul 16 11:10:11.245113 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.244790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748557646f-x82t4" event={"ID":"25144400-4cfe-4716-9145-c615ac45b250","Type":"ContainerStarted","Data":"005883ba807831da5406da117f2056a29b69ac7c6da0b77dfb0f078865829fc3"} Jul 16 11:10:11.274613 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:11.273129 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-748557646f-x82t4" podStartSLOduration=1.273106283 podStartE2EDuration="1.273106283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:11.272409519 +0000 UTC m=+95.038542655" watchObservedRunningTime="2026-07-16 11:10:11.273106283 +0000 UTC m=+95.039239421" Jul 16 11:10:12.252270 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:12.252227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb6t9" event={"ID":"5c5ed361-19c0-4de3-8efa-0ea720588bd4","Type":"ContainerStarted","Data":"bd8b7d25c8ef5cad34091c2a6f1467f5365a8bb5a62066637132eaedfef83b00"} Jul 16 11:10:12.252717 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:12.252304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb6t9" event={"ID":"5c5ed361-19c0-4de3-8efa-0ea720588bd4","Type":"ContainerStarted","Data":"bb36e915210d48ac6698e9d8dfb7a46a69cd0527a0b8e8075f4e47b507d759b8"} Jul 16 11:10:12.278813 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:12.278755 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kb6t9" podStartSLOduration=4.504150036 podStartE2EDuration="7.278734777s" totalImagesPullingTime="2.643582794s" totalInitContainerRuntime="131.001947ms" isStatefulPod=true podCreationTimestamp="2026-07-16 11:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:12.278206975 +0000 UTC m=+96.044340166" watchObservedRunningTime="2026-07-16 11:10:12.278734777 +0000 UTC m=+96.044867913" Jul 16 11:10:14.266227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:14.266177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms5rw" event={"ID":"46d57d98-2153-4a95-825e-53650d2e7435","Type":"ContainerStarted","Data":"25b272b69e669e9c9a5bc7546bb8244f219a01bf3b99f67af4160d2d47f7e233"} Jul 16 11:10:14.290140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:14.290085 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ms5rw" podStartSLOduration=33.982512461 podStartE2EDuration="1m2.290067658s" totalImagesPullingTime="28.307555197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:09:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:14.288592235 +0000 UTC m=+98.054725372" watchObservedRunningTime="2026-07-16 11:10:14.290067658 +0000 UTC m=+98.056200838" Jul 16 11:10:14.930021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:14.929967 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:14.981428 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:14.981365 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:10:15.003608 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.002715 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:10:15.003608 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.002855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075567 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.075894 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.075864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pzs\" (UniqueName: \"kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.176819 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.176782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.176938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.176855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.176938 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.176892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pzs\" (UniqueName: \"kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.177046 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.176951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.177046 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.176976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.177046 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.177008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.177185 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.177042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.178237 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.178189 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.178351 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.178242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.179054 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.179007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.179333 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.179310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.179910 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.179883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.180411 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.180350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.187840 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.187816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pzs\" (UniqueName: \"kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs\") pod \"console-6bd4b6ccdb-74278\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.271720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.271685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8"} Jul 16 11:10:15.314501 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.314315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:15.460975 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:15.460944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:10:15.463115 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:10:15.463094 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0458eadc_52a6_4e1d_95fb_e3c013ecd9ea.slice/crio-1f69aa85f91ba831627711a95230d8e60e0878adaa61fc6fdeafd2128a1a4b74 WatchSource:0}: Error finding container 1f69aa85f91ba831627711a95230d8e60e0878adaa61fc6fdeafd2128a1a4b74: Status 404 returned error can't find the container with id 1f69aa85f91ba831627711a95230d8e60e0878adaa61fc6fdeafd2128a1a4b74 Jul 16 11:10:16.278843 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.278801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e"} Jul 16 11:10:16.279439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.278883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a"} Jul 16 11:10:16.279439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.278922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3"} Jul 16 11:10:16.279439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.278961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77"} Jul 16 11:10:16.280447 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.280411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4b6ccdb-74278" event={"ID":"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea","Type":"ContainerStarted","Data":"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7"} Jul 16 11:10:16.280585 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.280467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4b6ccdb-74278" event={"ID":"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea","Type":"ContainerStarted","Data":"1f69aa85f91ba831627711a95230d8e60e0878adaa61fc6fdeafd2128a1a4b74"} Jul 16 11:10:16.299168 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:16.299112 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bd4b6ccdb-74278" podStartSLOduration=2.299097087 podStartE2EDuration="2.299097087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:16.298078907 +0000 UTC m=+100.064212044" watchObservedRunningTime="2026-07-16 11:10:16.299097087 +0000 UTC m=+100.065230224" Jul 16 11:10:17.287820 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:17.287723 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerStarted","Data":"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc"} Jul 16 11:10:17.315240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:17.315169 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8576553220000003 podStartE2EDuration="11.315149012s" totalImagesPullingTime="7.440435963s" totalInitContainerRuntime="17.057727ms" isStatefulPod=false podCreationTimestamp="2026-07-16 11:10:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:10:17.313789998 +0000 UTC m=+101.079923135" watchObservedRunningTime="2026-07-16 11:10:17.315149012 +0000 UTC m=+101.081282150" Jul 16 11:10:20.629038 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:20.628999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:22.137628 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:22.137593 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bhbv2" Jul 16 11:10:25.314555 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:25.314523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:25.315091 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:25.314592 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:25.319484 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:25.319462 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:26.131268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.131226 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84d9849dd8-gqcft" podUID="12cd9727-16e2-422c-bfa7-607cc282572a" containerName="console" containerID="cri-o://0d54a502faf2e251fa5ab7dba1a12a14855545e6b686f0207e8d810134a9df90" gracePeriod=15 Jul 16 11:10:26.330998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.330957 2580 generic.go:356] "Generic (PLEG): container finished" podID="12cd9727-16e2-422c-bfa7-607cc282572a" containerID="0d54a502faf2e251fa5ab7dba1a12a14855545e6b686f0207e8d810134a9df90" exitCode=0 Jul 16 11:10:26.331347 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.331019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9849dd8-gqcft" event={"ID":"12cd9727-16e2-422c-bfa7-607cc282572a","Type":"ContainerDied","Data":"0d54a502faf2e251fa5ab7dba1a12a14855545e6b686f0207e8d810134a9df90"} Jul 16 11:10:26.335863 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.335835 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:10:26.396276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.396207 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:10:26.475829 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.475778 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.475860 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.475899 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.475952 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.475990 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476224 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.476047 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zhj4\" (UniqueName: \"kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4\") pod \"12cd9727-16e2-422c-bfa7-607cc282572a\" (UID: \"12cd9727-16e2-422c-bfa7-607cc282572a\") " Jul 16 11:10:26.476278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.476256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config" (OuterVolumeSpecName: "console-config") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:26.476488 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.476467 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-console-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:26.478102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.478072 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:26.478539 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.478514 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca" (OuterVolumeSpecName: "service-ca") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:26.479259 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.479235 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:10:26.480367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.480344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4" (OuterVolumeSpecName: "kube-api-access-5zhj4") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "kube-api-access-5zhj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:10:26.480488 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.480364 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "12cd9727-16e2-422c-bfa7-607cc282572a" (UID: "12cd9727-16e2-422c-bfa7-607cc282572a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:10:26.577367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.577329 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhj4\" (UniqueName: \"kubernetes.io/projected/12cd9727-16e2-422c-bfa7-607cc282572a-kube-api-access-5zhj4\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:26.577367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.577363 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:26.577367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.577372 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12cd9727-16e2-422c-bfa7-607cc282572a-console-oauth-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:26.577694 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.577404 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-oauth-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:26.577694 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:26.577417 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd9727-16e2-422c-bfa7-607cc282572a-service-ca\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:27.336244 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:27.336206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9849dd8-gqcft" event={"ID":"12cd9727-16e2-422c-bfa7-607cc282572a","Type":"ContainerDied","Data":"11a7e4a3319c3c00998a02fba765ac2d7c42ab6b16adb280be0eb70355703a31"} Jul 16 11:10:27.336244 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:27.336230 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9849dd8-gqcft" Jul 16 11:10:27.336890 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:27.336275 2580 scope.go:117] "RemoveContainer" containerID="0d54a502faf2e251fa5ab7dba1a12a14855545e6b686f0207e8d810134a9df90" Jul 16 11:10:27.358472 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:27.358442 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:10:27.364625 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:27.364605 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d9849dd8-gqcft"] Jul 16 11:10:28.801642 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:28.801607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cd9727-16e2-422c-bfa7-607cc282572a" path="/var/lib/kubelet/pods/12cd9727-16e2-422c-bfa7-607cc282572a/volumes" Jul 16 11:10:37.942439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:37.942407 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgjd4_edac3d95-d82e-45ae-8063-fe14ecc387c5/serve-healthcheck-canary/0.log" Jul 16 11:10:39.953466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:39.953421 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-748557646f-x82t4" podUID="25144400-4cfe-4716-9145-c615ac45b250" containerName="console" containerID="cri-o://12a2d104bae39451d18a760a35556580ee80de0ea8719de5e00e2fb2d8f14cff" gracePeriod=15 Jul 16 11:10:40.387333 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.387301 2580 generic.go:356] "Generic (PLEG): container finished" podID="25144400-4cfe-4716-9145-c615ac45b250" containerID="12a2d104bae39451d18a760a35556580ee80de0ea8719de5e00e2fb2d8f14cff" exitCode=0 Jul 16 11:10:40.387520 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.387398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748557646f-x82t4" event={"ID":"25144400-4cfe-4716-9145-c615ac45b250","Type":"ContainerDied","Data":"12a2d104bae39451d18a760a35556580ee80de0ea8719de5e00e2fb2d8f14cff"} Jul 16 11:10:40.759153 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.759115 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:40.897683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897651 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8f58\" (UniqueName: \"kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897784 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.897866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.897852 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert\") pod \"25144400-4cfe-4716-9145-c615ac45b250\" (UID: \"25144400-4cfe-4716-9145-c615ac45b250\") " Jul 16 11:10:40.899048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.898426 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:40.899048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.898472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca" (OuterVolumeSpecName: "service-ca") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:40.899048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.898478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:40.899048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.898497 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config" (OuterVolumeSpecName: "console-config") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:10:40.899048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.898558 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-trusted-ca-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.904150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.904114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58" (OuterVolumeSpecName: "kube-api-access-z8f58") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "kube-api-access-z8f58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:10:40.904150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.904125 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:10:40.904321 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.904153 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "25144400-4cfe-4716-9145-c615ac45b250" (UID: "25144400-4cfe-4716-9145-c615ac45b250"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:10:40.999150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999115 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8f58\" (UniqueName: \"kubernetes.io/projected/25144400-4cfe-4716-9145-c615ac45b250-kube-api-access-z8f58\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.999150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999146 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-console-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.999150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999156 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-oauth-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.999614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999166 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25144400-4cfe-4716-9145-c615ac45b250-console-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.999614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999175 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-oauth-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:40.999614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:40.999183 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25144400-4cfe-4716-9145-c615ac45b250-service-ca\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:10:41.392298 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:41.392264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748557646f-x82t4" event={"ID":"25144400-4cfe-4716-9145-c615ac45b250","Type":"ContainerDied","Data":"005883ba807831da5406da117f2056a29b69ac7c6da0b77dfb0f078865829fc3"} Jul 16 11:10:41.392298 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:41.392284 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748557646f-x82t4" Jul 16 11:10:41.392568 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:41.392357 2580 scope.go:117] "RemoveContainer" containerID="12a2d104bae39451d18a760a35556580ee80de0ea8719de5e00e2fb2d8f14cff" Jul 16 11:10:41.412126 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:41.412094 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:41.415503 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:41.415476 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-748557646f-x82t4"] Jul 16 11:10:42.800855 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:10:42.800820 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25144400-4cfe-4716-9145-c615ac45b250" path="/var/lib/kubelet/pods/25144400-4cfe-4716-9145-c615ac45b250/volumes" Jul 16 11:11:24.250095 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250054 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250486 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cd9727-16e2-422c-bfa7-607cc282572a" containerName="console" Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250504 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cd9727-16e2-422c-bfa7-607cc282572a" containerName="console" Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250516 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25144400-4cfe-4716-9145-c615ac45b250" containerName="console" Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250524 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25144400-4cfe-4716-9145-c615ac45b250" containerName="console" Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250611 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25144400-4cfe-4716-9145-c615ac45b250" containerName="console" Jul 16 11:11:24.250697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.250625 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cd9727-16e2-422c-bfa7-607cc282572a" containerName="console" Jul 16 11:11:24.253212 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.253188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.266287 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.266263 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:11:24.332322 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332322 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.332550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.332527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcdx\" (UniqueName: \"kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.433631 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.433861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.433861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.433861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.433861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcdx\" (UniqueName: \"kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434080 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434080 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.433914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434295 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.434276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434432 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.434412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434542 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.434523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.434925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.434907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.436353 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.436328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.436353 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.436327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.441520 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.441498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcdx\" (UniqueName: \"kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx\") pod \"console-7fb58fd9bf-mjrfp\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.561776 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.561682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:24.702303 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:24.702270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:11:24.705643 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:11:24.705612 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a050615_1920_4d81_a90d_49ecb0978934.slice/crio-836b3bea1dff4a4d4281c6fe448a72a01b42fdd4c7f910f1c1824c8afe01a17a WatchSource:0}: Error finding container 836b3bea1dff4a4d4281c6fe448a72a01b42fdd4c7f910f1c1824c8afe01a17a: Status 404 returned error can't find the container with id 836b3bea1dff4a4d4281c6fe448a72a01b42fdd4c7f910f1c1824c8afe01a17a Jul 16 11:11:25.558692 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:25.558661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb58fd9bf-mjrfp" event={"ID":"1a050615-1920-4d81-a90d-49ecb0978934","Type":"ContainerStarted","Data":"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b"} Jul 16 11:11:25.559069 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:25.558711 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb58fd9bf-mjrfp" event={"ID":"1a050615-1920-4d81-a90d-49ecb0978934","Type":"ContainerStarted","Data":"836b3bea1dff4a4d4281c6fe448a72a01b42fdd4c7f910f1c1824c8afe01a17a"} Jul 16 11:11:25.575473 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:25.575397 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fb58fd9bf-mjrfp" podStartSLOduration=1.575367929 podStartE2EDuration="1.575367929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:11:25.574592896 +0000 UTC m=+169.340726033" watchObservedRunningTime="2026-07-16 11:11:25.575367929 +0000 UTC m=+169.341501065" Jul 16 11:11:25.843787 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:25.843701 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:26.565533 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565467 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="alertmanager" containerID="cri-o://e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8" gracePeriod=120 Jul 16 11:11:26.565998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565531 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-metric" containerID="cri-o://32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e" gracePeriod=120 Jul 16 11:11:26.565998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565567 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="prom-label-proxy" containerID="cri-o://c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc" gracePeriod=120 Jul 16 11:11:26.565998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565587 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy" containerID="cri-o://cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a" gracePeriod=120 Jul 16 11:11:26.565998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565619 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="config-reloader" containerID="cri-o://cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77" gracePeriod=120 Jul 16 11:11:26.565998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:26.565573 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-web" containerID="cri-o://6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3" gracePeriod=120 Jul 16 11:11:27.572462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572387 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc" exitCode=0 Jul 16 11:11:27.572462 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc"} Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e"} Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572417 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e" exitCode=0 Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572490 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a" exitCode=0 Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a"} Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77"} Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572496 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77" exitCode=0 Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572554 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8" exitCode=0 Jul 16 11:11:27.572849 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.572613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8"} Jul 16 11:11:27.802329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.802299 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:27.966145 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966090 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966156 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966191 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966217 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966247 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966272 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966764 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966403 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966764 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966436 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966764 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966466 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966764 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmkmr\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr\") pod \"23ddbebe-b156-4561-96d8-ca368cf52311\" (UID: \"23ddbebe-b156-4561-96d8-ca368cf52311\") " Jul 16 11:11:27.966764 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966637 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:11:27.967002 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966788 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:27.967002 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.966920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:11:27.967117 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.967014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:11:27.969429 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.969429 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969160 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.969429 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969183 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:11:27.969429 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume" (OuterVolumeSpecName: "config-volume") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.969667 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969459 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr" (OuterVolumeSpecName: "kube-api-access-pmkmr") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "kube-api-access-pmkmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:11:27.969939 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.969919 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.970224 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.970204 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.971089 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.971071 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out" (OuterVolumeSpecName: "config-out") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:11:27.974641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.974570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:27.980268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:27.980239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config" (OuterVolumeSpecName: "web-config") pod "23ddbebe-b156-4561-96d8-ca368cf52311" (UID: "23ddbebe-b156-4561-96d8-ca368cf52311"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:11:28.067969 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.067931 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-tls-assets\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.067969 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.067961 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.067969 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.067971 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-config-volume\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.067969 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.067979 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.067990 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068000 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23ddbebe-b156-4561-96d8-ca368cf52311-metrics-client-ca\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068009 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-alertmanager-main-db\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068017 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-secret-alertmanager-main-tls\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068025 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-cluster-tls-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068033 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23ddbebe-b156-4561-96d8-ca368cf52311-config-out\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068043 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23ddbebe-b156-4561-96d8-ca368cf52311-web-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.068222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.068051 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkmr\" (UniqueName: \"kubernetes.io/projected/23ddbebe-b156-4561-96d8-ca368cf52311-kube-api-access-pmkmr\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:11:28.578480 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.578447 2580 generic.go:356] "Generic (PLEG): container finished" podID="23ddbebe-b156-4561-96d8-ca368cf52311" containerID="6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3" exitCode=0 Jul 16 11:11:28.578826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.578501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3"} Jul 16 11:11:28.578826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.578518 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.578826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.578559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23ddbebe-b156-4561-96d8-ca368cf52311","Type":"ContainerDied","Data":"83c8a0862206f0f1a5ad70caef0563cdabe9ed667624a5ead6885dea2a4f60e8"} Jul 16 11:11:28.578826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.578591 2580 scope.go:117] "RemoveContainer" containerID="c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc" Jul 16 11:11:28.591782 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.591761 2580 scope.go:117] "RemoveContainer" containerID="32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e" Jul 16 11:11:28.598316 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.598298 2580 scope.go:117] "RemoveContainer" containerID="cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a" Jul 16 11:11:28.604909 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.604888 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:28.605071 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.605026 2580 scope.go:117] "RemoveContainer" containerID="6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3" Jul 16 11:11:28.611304 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.611282 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:28.611933 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.611908 2580 scope.go:117] "RemoveContainer" containerID="cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77" Jul 16 11:11:28.618527 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.618507 2580 scope.go:117] "RemoveContainer" containerID="e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8" Jul 16 11:11:28.624870 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.624853 2580 scope.go:117] "RemoveContainer" containerID="671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf" Jul 16 11:11:28.631499 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.631483 2580 scope.go:117] "RemoveContainer" containerID="c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc" Jul 16 11:11:28.631759 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.631739 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc\": container with ID starting with c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc not found: ID does not exist" containerID="c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc" Jul 16 11:11:28.631806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.631768 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc"} err="failed to get container status \"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc\": rpc error: code = NotFound desc = could not find container \"c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc\": container with ID starting with c62b518ebd3d4cc9dc3ab75c549a389267fa8c73ca147241759b7ae9af09cefc not found: ID does not exist" Jul 16 11:11:28.631806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.631803 2580 scope.go:117] "RemoveContainer" containerID="32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e" Jul 16 11:11:28.632026 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.632009 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e\": container with ID starting with 32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e not found: ID does not exist" containerID="32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e" Jul 16 11:11:28.632066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632031 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e"} err="failed to get container status \"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e\": rpc error: code = NotFound desc = could not find container \"32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e\": container with ID starting with 32d6c383679b977c1c0051848fdfffa8774657a796404cee4f0f86ad09d11c9e not found: ID does not exist" Jul 16 11:11:28.632066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632046 2580 scope.go:117] "RemoveContainer" containerID="cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a" Jul 16 11:11:28.632242 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.632225 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a\": container with ID starting with cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a not found: ID does not exist" containerID="cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a" Jul 16 11:11:28.632284 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632246 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a"} err="failed to get container status \"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a\": rpc error: code = NotFound desc = could not find container \"cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a\": container with ID starting with cbf3ab292d682b31bb53bdff285520ce45945c3b227e0172e6bdea9e78e1616a not found: ID does not exist" Jul 16 11:11:28.632284 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632261 2580 scope.go:117] "RemoveContainer" containerID="6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3" Jul 16 11:11:28.632500 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.632485 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3\": container with ID starting with 6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3 not found: ID does not exist" containerID="6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3" Jul 16 11:11:28.632555 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632501 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3"} err="failed to get container status \"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3\": rpc error: code = NotFound desc = could not find container \"6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3\": container with ID starting with 6fd4210b2f0634b69efb2b9ae6564fe9c02f30b90209cdb887b85aaf9f7019b3 not found: ID does not exist" Jul 16 11:11:28.632555 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632512 2580 scope.go:117] "RemoveContainer" containerID="cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77" Jul 16 11:11:28.632700 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.632683 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77\": container with ID starting with cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77 not found: ID does not exist" containerID="cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77" Jul 16 11:11:28.632733 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632705 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77"} err="failed to get container status \"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77\": rpc error: code = NotFound desc = could not find container \"cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77\": container with ID starting with cd688f1c0326187b3e2abbad06a6a3c6c7f660de2508f5993ce23917c5ddab77 not found: ID does not exist" Jul 16 11:11:28.632733 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632722 2580 scope.go:117] "RemoveContainer" containerID="e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8" Jul 16 11:11:28.632918 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.632902 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8\": container with ID starting with e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8 not found: ID does not exist" containerID="e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8" Jul 16 11:11:28.632956 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632921 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8"} err="failed to get container status \"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8\": rpc error: code = NotFound desc = could not find container \"e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8\": container with ID starting with e21054a89bf8054cb74471d5adcf89f6bbb1327da2ccfd3633d1f1adf5023ff8 not found: ID does not exist" Jul 16 11:11:28.632956 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.632934 2580 scope.go:117] "RemoveContainer" containerID="671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf" Jul 16 11:11:28.633115 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:11:28.633102 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf\": container with ID starting with 671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf not found: ID does not exist" containerID="671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf" Jul 16 11:11:28.633163 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.633117 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf"} err="failed to get container status \"671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf\": rpc error: code = NotFound desc = could not find container \"671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf\": container with ID starting with 671fcad58cb49177f637c58c6425f094e8a388c50bc0e810e5fdf4923da44bdf not found: ID does not exist" Jul 16 11:11:28.638218 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638193 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:28.638542 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638524 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="alertmanager" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638544 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="alertmanager" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638565 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="init-config-reloader" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638575 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="init-config-reloader" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638589 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="config-reloader" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638598 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="config-reloader" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638607 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-web" Jul 16 11:11:28.638624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638616 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-web" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638632 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="prom-label-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638640 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="prom-label-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638657 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638666 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638674 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-metric" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638682 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-metric" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638750 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="alertmanager" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638762 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-web" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638773 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="prom-label-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638784 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy-metric" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638794 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="kube-rbac-proxy" Jul 16 11:11:28.638948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.638803 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" containerName="config-reloader" Jul 16 11:11:28.642622 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.642603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.644789 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.644764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 11:11:28.645022 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 11:11:28.645152 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 11:11:28.645237 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645163 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 11:11:28.645302 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645280 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 11:11:28.645437 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645418 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sj7dp\"" Jul 16 11:11:28.645557 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645438 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 11:11:28.645557 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 11:11:28.645722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.645708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 11:11:28.650722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.650705 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 11:11:28.661561 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.661513 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:28.771770 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.771950 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.771950 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-volume\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.771950 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.771950 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnpp\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-kube-api-access-dqnpp\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.771981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772306 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-out\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.772306 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.772154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-web-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.801183 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.801148 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ddbebe-b156-4561-96d8-ca368cf52311" path="/var/lib/kubelet/pods/23ddbebe-b156-4561-96d8-ca368cf52311/volumes" Jul 16 11:11:28.872828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.872938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-out\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-web-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-volume\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873546 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873546 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnpp\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-kube-api-access-dqnpp\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.873546 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.873459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.875173 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.874133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.875173 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.874398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24b1558-ad87-4987-bf2d-787dfb72bb13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.875965 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.875927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.876449 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.876311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-out\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.876550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.876443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.876820 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.876795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.877042 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.877025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-web-config\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.877127 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.877105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.877166 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.877124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.877480 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.877462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-config-volume\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.877783 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.877769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e24b1558-ad87-4987-bf2d-787dfb72bb13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.882596 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.882571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnpp\" (UniqueName: \"kubernetes.io/projected/e24b1558-ad87-4987-bf2d-787dfb72bb13-kube-api-access-dqnpp\") pod \"alertmanager-main-0\" (UID: \"e24b1558-ad87-4987-bf2d-787dfb72bb13\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:28.952680 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:28.952644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 11:11:29.090563 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:29.090537 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 11:11:29.092700 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:11:29.092672 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24b1558_ad87_4987_bf2d_787dfb72bb13.slice/crio-761c2ea8e75316edd587406c40ec093f8f97ac4882d9b0b346d2facffbe7b4b6 WatchSource:0}: Error finding container 761c2ea8e75316edd587406c40ec093f8f97ac4882d9b0b346d2facffbe7b4b6: Status 404 returned error can't find the container with id 761c2ea8e75316edd587406c40ec093f8f97ac4882d9b0b346d2facffbe7b4b6 Jul 16 11:11:29.585232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:29.585151 2580 generic.go:356] "Generic (PLEG): container finished" podID="e24b1558-ad87-4987-bf2d-787dfb72bb13" containerID="a1f3ec1e122fe17544355efdfe30dd36f757d82ff340e3517221fe224c060fa0" exitCode=0 Jul 16 11:11:29.585232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:29.585198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerDied","Data":"a1f3ec1e122fe17544355efdfe30dd36f757d82ff340e3517221fe224c060fa0"} Jul 16 11:11:29.585771 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:29.585262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"761c2ea8e75316edd587406c40ec093f8f97ac4882d9b0b346d2facffbe7b4b6"} Jul 16 11:11:30.592153 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"cf3d427636d22f74158088627e9e099326d13827b4501c64c87ebf330e4b3534"} Jul 16 11:11:30.592644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"09558d61f9c38d96b5adb14e2a036a88d2cc35c0fefb19b1eb3c9b887c37001a"} Jul 16 11:11:30.592644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"1981666a48d0f830706a3e644dd72e75d83cc6c8e93b59c01df35e7d4cd550df"} Jul 16 11:11:30.592644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"f8bef280975f89df6d9c8594e3725a3a7fe40fa9b8741358a3dcf33c25058b72"} Jul 16 11:11:30.592644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"ed422dd01dcb330ade91d924a1d8d2cf5740bdbae1b03fbd70fb5d078ed92e11"} Jul 16 11:11:30.592644 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.592310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e24b1558-ad87-4987-bf2d-787dfb72bb13","Type":"ContainerStarted","Data":"be8001c44dde3d288ff51040b5940f2dcfeb7ffca4f4bebec155f30b1346eeff"} Jul 16 11:11:30.622457 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:30.622405 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.606291825 podStartE2EDuration="2.622372702s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.080877ms" isStatefulPod=false podCreationTimestamp="2026-07-16 11:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:11:30.620591635 +0000 UTC m=+174.386724801" watchObservedRunningTime="2026-07-16 11:11:30.622372702 +0000 UTC m=+174.388505838" Jul 16 11:11:34.562481 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:34.562446 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:34.562855 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:34.562503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:34.567293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:34.567270 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:34.611194 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:34.611169 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:11:34.664985 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:34.664956 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:11:59.686974 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:59.686905 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bd4b6ccdb-74278" podUID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" containerName="console" containerID="cri-o://de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7" gracePeriod=15 Jul 16 11:11:59.928195 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:11:59.928171 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:12:00.018188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018099 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018152 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018180 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018236 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6pzs\" (UniqueName: \"kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018270 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle\") pod \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\" (UID: \"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea\") " Jul 16 11:12:00.018673 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018584 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:12:00.018845 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018814 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config" (OuterVolumeSpecName: "console-config") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:12:00.018845 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:12:00.018944 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.018859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:12:00.020310 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.020281 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:12:00.020697 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.020683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:12:00.020752 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.020701 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs" (OuterVolumeSpecName: "kube-api-access-h6pzs") pod "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" (UID: "0458eadc-52a6-4e1d-95fb-e3c013ecd9ea"). InnerVolumeSpecName "kube-api-access-h6pzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:12:00.119683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119649 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119679 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-service-ca\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119688 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119696 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pzs\" (UniqueName: \"kubernetes.io/projected/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-kube-api-access-h6pzs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119705 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-oauth-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119713 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-trusted-ca-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.119899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.119721 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea-console-oauth-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:12:00.702583 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.702544 2580 generic.go:356] "Generic (PLEG): container finished" podID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" containerID="de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7" exitCode=0 Jul 16 11:12:00.703053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.702597 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4b6ccdb-74278" Jul 16 11:12:00.703053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.702634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4b6ccdb-74278" event={"ID":"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea","Type":"ContainerDied","Data":"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7"} Jul 16 11:12:00.703053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.702735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4b6ccdb-74278" event={"ID":"0458eadc-52a6-4e1d-95fb-e3c013ecd9ea","Type":"ContainerDied","Data":"1f69aa85f91ba831627711a95230d8e60e0878adaa61fc6fdeafd2128a1a4b74"} Jul 16 11:12:00.703053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.702767 2580 scope.go:117] "RemoveContainer" containerID="de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7" Jul 16 11:12:00.711168 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.711153 2580 scope.go:117] "RemoveContainer" containerID="de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7" Jul 16 11:12:00.711442 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:12:00.711423 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7\": container with ID starting with de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7 not found: ID does not exist" containerID="de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7" Jul 16 11:12:00.711515 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.711455 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7"} err="failed to get container status \"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7\": rpc error: code = NotFound desc = could not find container \"de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7\": container with ID starting with de249875b4a6650e6b57f47fb68707ea6e06e6319a1aa8464bb8748176ec3ef7 not found: ID does not exist" Jul 16 11:12:00.722531 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.722505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:12:00.725556 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.725536 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bd4b6ccdb-74278"] Jul 16 11:12:00.801225 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:00.801192 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" path="/var/lib/kubelet/pods/0458eadc-52a6-4e1d-95fb-e3c013ecd9ea/volumes" Jul 16 11:12:30.959552 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.959519 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr"] Jul 16 11:12:30.960066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.959820 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" containerName="console" Jul 16 11:12:30.960066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.959833 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" containerName="console" Jul 16 11:12:30.960066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.959877 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0458eadc-52a6-4e1d-95fb-e3c013ecd9ea" containerName="console" Jul 16 11:12:30.963114 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.963094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:30.965211 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.965190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 11:12:30.966221 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.966198 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2v7jm\"" Jul 16 11:12:30.966221 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.966204 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 11:12:30.966395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.966204 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 11:12:30.966395 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.966261 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 11:12:30.971129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:30.971109 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr"] Jul 16 11:12:31.053483 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.053454 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829"] Jul 16 11:12:31.055753 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.055733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.058073 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.058057 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 11:12:31.058539 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.058512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89bd671b-2983-4766-b964-af764e797d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.058598 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.058567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k296z\" (UniqueName: \"kubernetes.io/projected/89bd671b-2983-4766-b964-af764e797d12-kube-api-access-k296z\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.069105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.069079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829"] Jul 16 11:12:31.159897 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.159855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/850953e7-2744-4a28-b363-81d9076e12cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.160074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.159941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89bd671b-2983-4766-b964-af764e797d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.160074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.159980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k296z\" (UniqueName: \"kubernetes.io/projected/89bd671b-2983-4766-b964-af764e797d12-kube-api-access-k296z\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.160074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.160004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850953e7-2744-4a28-b363-81d9076e12cd-tmp\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.160257 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.160244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpnw2\" (UniqueName: \"kubernetes.io/projected/850953e7-2744-4a28-b363-81d9076e12cd-kube-api-access-vpnw2\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.162526 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.162503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89bd671b-2983-4766-b964-af764e797d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.167502 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.167465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k296z\" (UniqueName: \"kubernetes.io/projected/89bd671b-2983-4766-b964-af764e797d12-kube-api-access-k296z\") pod \"managed-serviceaccount-addon-agent-7d579ddc46-t6lvr\" (UID: \"89bd671b-2983-4766-b964-af764e797d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.260826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.260718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850953e7-2744-4a28-b363-81d9076e12cd-tmp\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.260826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.260776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpnw2\" (UniqueName: \"kubernetes.io/projected/850953e7-2744-4a28-b363-81d9076e12cd-kube-api-access-vpnw2\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.260826 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.260821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/850953e7-2744-4a28-b363-81d9076e12cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.261196 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.261168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850953e7-2744-4a28-b363-81d9076e12cd-tmp\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.263293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.263273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/850953e7-2744-4a28-b363-81d9076e12cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.270309 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.270284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpnw2\" (UniqueName: \"kubernetes.io/projected/850953e7-2744-4a28-b363-81d9076e12cd-kube-api-access-vpnw2\") pod \"klusterlet-addon-workmgr-6666fb5f4-lv829\" (UID: \"850953e7-2744-4a28-b363-81d9076e12cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.289264 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.289227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" Jul 16 11:12:31.365363 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.365331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:31.415795 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.415709 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr"] Jul 16 11:12:31.419711 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:12:31.419666 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89bd671b_2983_4766_b964_af764e797d12.slice/crio-ea6123e880a8cc93a371728744f342a7be17b9d2df2788e6643a1381c29a9677 WatchSource:0}: Error finding container ea6123e880a8cc93a371728744f342a7be17b9d2df2788e6643a1381c29a9677: Status 404 returned error can't find the container with id ea6123e880a8cc93a371728744f342a7be17b9d2df2788e6643a1381c29a9677 Jul 16 11:12:31.498618 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.498587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829"] Jul 16 11:12:31.501391 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:12:31.501351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850953e7_2744_4a28_b363_81d9076e12cd.slice/crio-d0903ab3cf9521ed0c51ca88ada93b26a720603f98ab1027cef11798982a4b0a WatchSource:0}: Error finding container d0903ab3cf9521ed0c51ca88ada93b26a720603f98ab1027cef11798982a4b0a: Status 404 returned error can't find the container with id d0903ab3cf9521ed0c51ca88ada93b26a720603f98ab1027cef11798982a4b0a Jul 16 11:12:31.811603 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.811565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" event={"ID":"850953e7-2744-4a28-b363-81d9076e12cd","Type":"ContainerStarted","Data":"d0903ab3cf9521ed0c51ca88ada93b26a720603f98ab1027cef11798982a4b0a"} Jul 16 11:12:31.812645 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:31.812623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" event={"ID":"89bd671b-2983-4766-b964-af764e797d12","Type":"ContainerStarted","Data":"ea6123e880a8cc93a371728744f342a7be17b9d2df2788e6643a1381c29a9677"} Jul 16 11:12:34.829247 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:34.829212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" event={"ID":"89bd671b-2983-4766-b964-af764e797d12","Type":"ContainerStarted","Data":"c9ff991628d284ae606727f29eaca561b0e2739c4fee03d8c15c57a2d7fe7280"} Jul 16 11:12:34.847604 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:34.847538 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d579ddc46-t6lvr" podStartSLOduration=2.370417058 podStartE2EDuration="4.847517344s" totalImagesPullingTime="2.477100286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:12:34.84560049 +0000 UTC m=+238.611733629" watchObservedRunningTime="2026-07-16 11:12:34.847517344 +0000 UTC m=+238.613650483" Jul 16 11:12:35.834278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:35.834229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" event={"ID":"850953e7-2744-4a28-b363-81d9076e12cd","Type":"ContainerStarted","Data":"3e30ebf7f955e9e4b5e9bb1f340c21ca78476cf7fbdcb6768f870edb53d1ecac"} Jul 16 11:12:35.834733 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:35.834521 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:35.835913 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:35.835891 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" Jul 16 11:12:35.851125 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:35.851052 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6666fb5f4-lv829" podStartSLOduration=1.131881246 podStartE2EDuration="4.851037681s" totalImagesPullingTime="3.719156435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:12:35.850609916 +0000 UTC m=+239.616743067" watchObservedRunningTime="2026-07-16 11:12:35.851037681 +0000 UTC m=+239.617170864" Jul 16 11:12:51.451818 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.451782 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k"] Jul 16 11:12:51.457082 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.457052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.460061 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.460038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 11:12:51.460818 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.460800 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 11:12:51.460916 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.460808 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4qn48\"" Jul 16 11:12:51.466729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.466703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k"] Jul 16 11:12:51.520660 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.520630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.520875 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.520673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf59x\" (UniqueName: \"kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.520875 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.520695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.621424 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.621357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.621620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.621440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf59x\" (UniqueName: \"kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.621620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.621471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.621752 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.621738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.621863 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.621837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.630691 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.630621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf59x\" (UniqueName: \"kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.766978 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.766887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:12:51.898582 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:51.898557 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k"] Jul 16 11:12:51.901200 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:12:51.901174 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a86f24d_38bc_4313_b57a_8d4a5a8001f3.slice/crio-38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d WatchSource:0}: Error finding container 38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d: Status 404 returned error can't find the container with id 38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d Jul 16 11:12:52.899536 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:52.899499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" event={"ID":"7a86f24d-38bc-4313-b57a-8d4a5a8001f3","Type":"ContainerStarted","Data":"38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d"} Jul 16 11:12:58.924426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:58.924359 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerID="7db0dd396dbcbb5fc9c4c017e2e5346e28ab3622b7ae0912f8ab396e74c71d86" exitCode=0 Jul 16 11:12:58.924801 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:12:58.924450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" event={"ID":"7a86f24d-38bc-4313-b57a-8d4a5a8001f3","Type":"ContainerDied","Data":"7db0dd396dbcbb5fc9c4c017e2e5346e28ab3622b7ae0912f8ab396e74c71d86"} Jul 16 11:13:01.938906 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:01.938870 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerID="80e95beec099b1c08e39c6e4d63c8b6a70985c88dc240bf78990b5efab417964" exitCode=0 Jul 16 11:13:01.939371 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:01.938961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" event={"ID":"7a86f24d-38bc-4313-b57a-8d4a5a8001f3","Type":"ContainerDied","Data":"80e95beec099b1c08e39c6e4d63c8b6a70985c88dc240bf78990b5efab417964"} Jul 16 11:13:10.975622 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:10.975586 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerID="0003ac1b9132aee46963920538fc64f6b59587cdeed90a131e4b52c5203f5e53" exitCode=0 Jul 16 11:13:10.975995 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:10.975668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" event={"ID":"7a86f24d-38bc-4313-b57a-8d4a5a8001f3","Type":"ContainerDied","Data":"0003ac1b9132aee46963920538fc64f6b59587cdeed90a131e4b52c5203f5e53"} Jul 16 11:13:12.101518 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.101492 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:13:12.206263 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.206233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf59x\" (UniqueName: \"kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x\") pod \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " Jul 16 11:13:12.206457 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.206294 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle\") pod \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " Jul 16 11:13:12.206457 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.206333 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util\") pod \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\" (UID: \"7a86f24d-38bc-4313-b57a-8d4a5a8001f3\") " Jul 16 11:13:12.206988 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.206957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle" (OuterVolumeSpecName: "bundle") pod "7a86f24d-38bc-4313-b57a-8d4a5a8001f3" (UID: "7a86f24d-38bc-4313-b57a-8d4a5a8001f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:12.208456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.208438 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x" (OuterVolumeSpecName: "kube-api-access-pf59x") pod "7a86f24d-38bc-4313-b57a-8d4a5a8001f3" (UID: "7a86f24d-38bc-4313-b57a-8d4a5a8001f3"). InnerVolumeSpecName "kube-api-access-pf59x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:13:12.210871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.210847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util" (OuterVolumeSpecName: "util") pod "7a86f24d-38bc-4313-b57a-8d4a5a8001f3" (UID: "7a86f24d-38bc-4313-b57a-8d4a5a8001f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:12.307095 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.306994 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:12.307095 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.307039 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:12.307095 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.307055 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf59x\" (UniqueName: \"kubernetes.io/projected/7a86f24d-38bc-4313-b57a-8d4a5a8001f3-kube-api-access-pf59x\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:12.985767 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.985730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" event={"ID":"7a86f24d-38bc-4313-b57a-8d4a5a8001f3","Type":"ContainerDied","Data":"38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d"} Jul 16 11:13:12.985925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.985791 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38def1475ec9c864e3c6db973f95348839f2431e08753bfa31399331f60e439d" Jul 16 11:13:12.985925 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:12.985742 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrq2k" Jul 16 11:13:24.698451 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698415 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw"] Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698695 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="pull" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698705 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="pull" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698718 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="util" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698724 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="util" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698733 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="extract" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698739 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="extract" Jul 16 11:13:24.698828 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.698787 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a86f24d-38bc-4313-b57a-8d4a5a8001f3" containerName="extract" Jul 16 11:13:24.701003 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.700985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.714579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.714556 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 11:13:24.719315 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.719290 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 11:13:24.719596 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.719579 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4qn48\"" Jul 16 11:13:24.812913 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.812883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.813099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.812976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntms\" (UniqueName: \"kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.813099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.813005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.819767 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.819738 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw"] Jul 16 11:13:24.913602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.913560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntms\" (UniqueName: \"kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.913602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.913605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.913871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.913716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.914052 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.914030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.914099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.914082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:24.933698 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:24.933663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntms\" (UniqueName: \"kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:25.009616 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:25.009539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:25.160774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:25.160741 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw"] Jul 16 11:13:25.173023 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:13:25.172997 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25371b3_328c_486a_856d_33937f1a2352.slice/crio-bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca WatchSource:0}: Error finding container bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca: Status 404 returned error can't find the container with id bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca Jul 16 11:13:26.041187 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:26.041152 2580 generic.go:356] "Generic (PLEG): container finished" podID="f25371b3-328c-486a-856d-33937f1a2352" containerID="20c8cd07432e1037fa602114fe9df384fbfd6aaf64169a8214d35a0c3baa9266" exitCode=0 Jul 16 11:13:26.041653 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:26.041248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" event={"ID":"f25371b3-328c-486a-856d-33937f1a2352","Type":"ContainerDied","Data":"20c8cd07432e1037fa602114fe9df384fbfd6aaf64169a8214d35a0c3baa9266"} Jul 16 11:13:26.041653 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:26.041299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" event={"ID":"f25371b3-328c-486a-856d-33937f1a2352","Type":"ContainerStarted","Data":"bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca"} Jul 16 11:13:29.054214 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:29.054179 2580 generic.go:356] "Generic (PLEG): container finished" podID="f25371b3-328c-486a-856d-33937f1a2352" containerID="549ca0eb0b789a3545365562c11732ad76e77426622300d8543a569242be6074" exitCode=0 Jul 16 11:13:29.054631 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:29.054248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" event={"ID":"f25371b3-328c-486a-856d-33937f1a2352","Type":"ContainerDied","Data":"549ca0eb0b789a3545365562c11732ad76e77426622300d8543a569242be6074"} Jul 16 11:13:30.060231 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:30.060198 2580 generic.go:356] "Generic (PLEG): container finished" podID="f25371b3-328c-486a-856d-33937f1a2352" containerID="61869fdc2d679eb3234060d3673c6ac899750c85351984aee4ef7be0ce0cfacf" exitCode=0 Jul 16 11:13:30.060631 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:30.060293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" event={"ID":"f25371b3-328c-486a-856d-33937f1a2352","Type":"ContainerDied","Data":"61869fdc2d679eb3234060d3673c6ac899750c85351984aee4ef7be0ce0cfacf"} Jul 16 11:13:31.196241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.196217 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:31.371116 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.371091 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zntms\" (UniqueName: \"kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms\") pod \"f25371b3-328c-486a-856d-33937f1a2352\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " Jul 16 11:13:31.371282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.371140 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle\") pod \"f25371b3-328c-486a-856d-33937f1a2352\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " Jul 16 11:13:31.371282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.371167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util\") pod \"f25371b3-328c-486a-856d-33937f1a2352\" (UID: \"f25371b3-328c-486a-856d-33937f1a2352\") " Jul 16 11:13:31.371499 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.371476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle" (OuterVolumeSpecName: "bundle") pod "f25371b3-328c-486a-856d-33937f1a2352" (UID: "f25371b3-328c-486a-856d-33937f1a2352"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:31.373200 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.373172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms" (OuterVolumeSpecName: "kube-api-access-zntms") pod "f25371b3-328c-486a-856d-33937f1a2352" (UID: "f25371b3-328c-486a-856d-33937f1a2352"). InnerVolumeSpecName "kube-api-access-zntms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:13:31.376221 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.376188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util" (OuterVolumeSpecName: "util") pod "f25371b3-328c-486a-856d-33937f1a2352" (UID: "f25371b3-328c-486a-856d-33937f1a2352"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:31.471732 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.471696 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntms\" (UniqueName: \"kubernetes.io/projected/f25371b3-328c-486a-856d-33937f1a2352-kube-api-access-zntms\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:31.471732 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.471725 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:31.471732 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.471738 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25371b3-328c-486a-856d-33937f1a2352-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.975742 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h"] Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976357 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="util" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976399 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="util" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976426 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="pull" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976436 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="pull" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976456 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="extract" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976464 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="extract" Jul 16 11:13:31.977928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:31.976578 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25371b3-328c-486a-856d-33937f1a2352" containerName="extract" Jul 16 11:13:32.011285 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.011243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h"] Jul 16 11:13:32.011482 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.011366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.013967 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.013948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g6fxv\"" Jul 16 11:13:32.014723 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.014707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 11:13:32.014723 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.014713 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 11:13:32.069472 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.069445 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" Jul 16 11:13:32.069472 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.069466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flbkpw" event={"ID":"f25371b3-328c-486a-856d-33937f1a2352","Type":"ContainerDied","Data":"bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca"} Jul 16 11:13:32.069664 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.069507 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc71b0d8df1dd6906a94d1265a0ee270b01e8002830e46490f784cb75cdf7ca" Jul 16 11:13:32.181072 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.181036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st96c\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-kube-api-access-st96c\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.181072 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.181072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.281997 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.281919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st96c\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-kube-api-access-st96c\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.281997 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.281952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.298587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.298547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.299061 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.299031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st96c\" (UniqueName: \"kubernetes.io/projected/cdab9ee3-1e31-48b4-9470-9872373fe571-kube-api-access-st96c\") pod \"cert-manager-cainjector-559cdd8c87-4wm5h\" (UID: \"cdab9ee3-1e31-48b4-9470-9872373fe571\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.320908 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.320877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" Jul 16 11:13:32.490208 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:32.490171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h"] Jul 16 11:13:32.493072 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:13:32.493040 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdab9ee3_1e31_48b4_9470_9872373fe571.slice/crio-9485bfea47d5916004aa65c1cbb658c78d17b2d35c55cbbc839635b8936a3105 WatchSource:0}: Error finding container 9485bfea47d5916004aa65c1cbb658c78d17b2d35c55cbbc839635b8936a3105: Status 404 returned error can't find the container with id 9485bfea47d5916004aa65c1cbb658c78d17b2d35c55cbbc839635b8936a3105 Jul 16 11:13:33.075593 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:33.075547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" event={"ID":"cdab9ee3-1e31-48b4-9470-9872373fe571","Type":"ContainerStarted","Data":"9485bfea47d5916004aa65c1cbb658c78d17b2d35c55cbbc839635b8936a3105"} Jul 16 11:13:35.087173 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:35.087137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" event={"ID":"cdab9ee3-1e31-48b4-9470-9872373fe571","Type":"ContainerStarted","Data":"7a2eb89a86ad379fc71e2dae4c14ea49c3d5e0a97723e527203dd313edc8245d"} Jul 16 11:13:35.123039 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:35.122986 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wm5h" podStartSLOduration=1.6604471969999999 podStartE2EDuration="4.122968519s" totalImagesPullingTime="2.462521322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:13:35.122358629 +0000 UTC m=+298.888491767" watchObservedRunningTime="2026-07-16 11:13:35.122968519 +0000 UTC m=+298.889101654" Jul 16 11:13:36.625452 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:36.625424 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:13:36.625452 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:36.625442 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:13:36.637980 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:36.637958 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 11:13:42.816791 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.816754 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8"] Jul 16 11:13:42.834885 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.834862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.841152 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.840730 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 11:13:42.841269 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.841250 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4qn48\"" Jul 16 11:13:42.841342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.841286 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 11:13:42.845210 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.845189 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8"] Jul 16 11:13:42.851805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.851784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.851900 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.851833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdvq\" (UniqueName: \"kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.851940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.851890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.953057 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.953007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.953227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.953097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.953227 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.953142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdvq\" (UniqueName: \"kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.953435 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.953414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.953517 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.953497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:42.965112 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:42.965088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdvq\" (UniqueName: \"kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:43.144008 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:43.143977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:43.290262 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:43.290237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8"] Jul 16 11:13:43.292344 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:13:43.292315 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff22ec7_dcb2_4fb6_b2c3_16ee59650244.slice/crio-694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5 WatchSource:0}: Error finding container 694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5: Status 404 returned error can't find the container with id 694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5 Jul 16 11:13:44.123560 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:44.123528 2580 generic.go:356] "Generic (PLEG): container finished" podID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerID="3825d19f8a2e85cadbc7b8e1602b1cf71d848099582b7f26e8fd517c8c50dd5e" exitCode=0 Jul 16 11:13:44.123948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:44.123613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" event={"ID":"cff22ec7-dcb2-4fb6-b2c3-16ee59650244","Type":"ContainerDied","Data":"3825d19f8a2e85cadbc7b8e1602b1cf71d848099582b7f26e8fd517c8c50dd5e"} Jul 16 11:13:44.123948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:44.123665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" event={"ID":"cff22ec7-dcb2-4fb6-b2c3-16ee59650244","Type":"ContainerStarted","Data":"694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5"} Jul 16 11:13:44.124029 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:44.124022 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:13:45.130126 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:45.130097 2580 generic.go:356] "Generic (PLEG): container finished" podID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerID="58bfa378b9f2a4db1a3cdb9939d344c4dc468cde6e72d66ecedad60e33b06537" exitCode=0 Jul 16 11:13:45.130534 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:45.130142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" event={"ID":"cff22ec7-dcb2-4fb6-b2c3-16ee59650244","Type":"ContainerDied","Data":"58bfa378b9f2a4db1a3cdb9939d344c4dc468cde6e72d66ecedad60e33b06537"} Jul 16 11:13:46.135719 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:46.135679 2580 generic.go:356] "Generic (PLEG): container finished" podID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerID="82433a9f397d88580edcdc441a9b4bfdce1de1ecb3aab4af5e3d7ba53d128a84" exitCode=0 Jul 16 11:13:46.136170 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:46.135733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" event={"ID":"cff22ec7-dcb2-4fb6-b2c3-16ee59650244","Type":"ContainerDied","Data":"82433a9f397d88580edcdc441a9b4bfdce1de1ecb3aab4af5e3d7ba53d128a84"} Jul 16 11:13:47.259983 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.259960 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:47.285712 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.285677 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrdvq\" (UniqueName: \"kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq\") pod \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " Jul 16 11:13:47.285866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.285730 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle\") pod \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " Jul 16 11:13:47.285866 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.285764 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util\") pod \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\" (UID: \"cff22ec7-dcb2-4fb6-b2c3-16ee59650244\") " Jul 16 11:13:47.286673 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.286648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle" (OuterVolumeSpecName: "bundle") pod "cff22ec7-dcb2-4fb6-b2c3-16ee59650244" (UID: "cff22ec7-dcb2-4fb6-b2c3-16ee59650244"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:47.287698 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.287664 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq" (OuterVolumeSpecName: "kube-api-access-nrdvq") pod "cff22ec7-dcb2-4fb6-b2c3-16ee59650244" (UID: "cff22ec7-dcb2-4fb6-b2c3-16ee59650244"). InnerVolumeSpecName "kube-api-access-nrdvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:13:47.291760 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.291741 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util" (OuterVolumeSpecName: "util") pod "cff22ec7-dcb2-4fb6-b2c3-16ee59650244" (UID: "cff22ec7-dcb2-4fb6-b2c3-16ee59650244"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:13:47.386872 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.386838 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:47.386872 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.386868 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdvq\" (UniqueName: \"kubernetes.io/projected/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-kube-api-access-nrdvq\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:47.386872 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:47.386881 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cff22ec7-dcb2-4fb6-b2c3-16ee59650244-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:13:48.144747 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:48.144710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" event={"ID":"cff22ec7-dcb2-4fb6-b2c3-16ee59650244","Type":"ContainerDied","Data":"694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5"} Jul 16 11:13:48.144747 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:48.144733 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affpfl8" Jul 16 11:13:48.144959 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:48.144764 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694df6d53c0e8ab4488cc69687a403f370e1630dce9dc3162a2a8db52adc90a5" Jul 16 11:13:55.339899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.339865 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j"] Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340130 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="extract" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="extract" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340160 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="pull" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340165 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="pull" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340173 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="util" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340178 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="util" Jul 16 11:13:55.340268 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.340230 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff22ec7-dcb2-4fb6-b2c3-16ee59650244" containerName="extract" Jul 16 11:13:55.342336 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.342321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.348358 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.348334 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 11:13:55.349321 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.349298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 11:13:55.350016 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.349999 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8lh67\"" Jul 16 11:13:55.350016 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.350001 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 11:13:55.351750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.351726 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 11:13:55.394760 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.394721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j"] Jul 16 11:13:55.447363 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.447321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.447586 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.447372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-manifests\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.447586 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.447485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.447586 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.447534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/829b6646-37b1-48b7-9806-870dd3a62cd6-kube-api-access-6s894\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.447701 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.447600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-tmp\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548356 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-manifests\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548574 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548574 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/829b6646-37b1-48b7-9806-870dd3a62cd6-kube-api-access-6s894\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548574 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-tmp\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548574 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.548800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.548759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-manifests\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.550831 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.550798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829b6646-37b1-48b7-9806-870dd3a62cd6-tmp\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.550961 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.550877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.551026 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.550959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/829b6646-37b1-48b7-9806-870dd3a62cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.559205 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.559179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/829b6646-37b1-48b7-9806-870dd3a62cd6-kube-api-access-6s894\") pod \"opendatahub-operator-controller-manager-5f8c4d5c89-drz7j\" (UID: \"829b6646-37b1-48b7-9806-870dd3a62cd6\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.652193 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.652151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:13:55.786481 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:55.786446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j"] Jul 16 11:13:55.790015 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:13:55.789993 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829b6646_37b1_48b7_9806_870dd3a62cd6.slice/crio-47e2e878c5ae2fdad60fbc57bd6da0446db4e5861bf42e32c361fcd9a771a051 WatchSource:0}: Error finding container 47e2e878c5ae2fdad60fbc57bd6da0446db4e5861bf42e32c361fcd9a771a051: Status 404 returned error can't find the container with id 47e2e878c5ae2fdad60fbc57bd6da0446db4e5861bf42e32c361fcd9a771a051 Jul 16 11:13:56.176298 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:56.176265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" event={"ID":"829b6646-37b1-48b7-9806-870dd3a62cd6","Type":"ContainerStarted","Data":"47e2e878c5ae2fdad60fbc57bd6da0446db4e5861bf42e32c361fcd9a771a051"} Jul 16 11:13:59.194856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:13:59.194818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" event={"ID":"829b6646-37b1-48b7-9806-870dd3a62cd6","Type":"ContainerStarted","Data":"aa7690ba293cfe22b46f232d7015a98f1cb42280986c9ea833660098444ada30"} Jul 16 11:14:00.200242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:00.200206 2580 generic.go:356] "Generic (PLEG): container finished" podID="829b6646-37b1-48b7-9806-870dd3a62cd6" containerID="aa7690ba293cfe22b46f232d7015a98f1cb42280986c9ea833660098444ada30" exitCode=0 Jul 16 11:14:00.200651 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:00.200259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" event={"ID":"829b6646-37b1-48b7-9806-870dd3a62cd6","Type":"ContainerDied","Data":"aa7690ba293cfe22b46f232d7015a98f1cb42280986c9ea833660098444ada30"} Jul 16 11:14:01.206920 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:01.206883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" event={"ID":"829b6646-37b1-48b7-9806-870dd3a62cd6","Type":"ContainerStarted","Data":"aeb2082b98fdb5875472ca39c8987729c4c5d5df1f3beaf8f1de0be49134d9e4"} Jul 16 11:14:01.207438 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:01.207005 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:14:01.234839 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:01.234775 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" podStartSLOduration=1.945990763 podStartE2EDuration="6.23475492s" totalImagesPullingTime="3.225484198s" totalInitContainerRuntime="1.063279959s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:13:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:14:01.233307545 +0000 UTC m=+324.999440683" watchObservedRunningTime="2026-07-16 11:14:01.23475492 +0000 UTC m=+325.000888058" Jul 16 11:14:12.213474 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:12.213440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8c4d5c89-drz7j" Jul 16 11:14:15.922396 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.922351 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg"] Jul 16 11:14:15.930505 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.930477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:15.934288 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.934264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 11:14:15.936541 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.936520 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 11:14:15.941937 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.941890 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 11:14:15.942060 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.942036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 11:14:15.981498 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:15.981464 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg"] Jul 16 11:14:16.023622 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.023584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tmp\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.023813 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.023717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.023813 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.023752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlnz\" (UniqueName: \"kubernetes.io/projected/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-kube-api-access-frlnz\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.046934 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.046898 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw"] Jul 16 11:14:16.050937 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.050907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.062792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.062755 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 11:14:16.062951 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.062763 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 11:14:16.081290 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.081266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 11:14:16.081290 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.081282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 11:14:16.081548 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.081371 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 11:14:16.081614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.081599 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hxptp\"" Jul 16 11:14:16.099537 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.099514 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw"] Jul 16 11:14:16.124308 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tmp\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.124506 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-metrics-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.124506 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njch\" (UniqueName: \"kubernetes.io/projected/da6efd8b-3ef2-4346-808a-f6216d0ac660-kube-api-access-6njch\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.124626 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da6efd8b-3ef2-4346-808a-f6216d0ac660-manager-config\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.124626 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.124721 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlnz\" (UniqueName: \"kubernetes.io/projected/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-kube-api-access-frlnz\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.124721 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.124670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.124721 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:14:16.124700 2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 16 11:14:16.124857 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:14:16.124782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs podName:4e0f2523-eb5b-4fa4-a7df-769a9c01aae5 nodeName:}" failed. No retries permitted until 2026-07-16 11:14:16.624760073 +0000 UTC m=+340.390893195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs") pod "kube-auth-proxy-79cdcbc7df-w8bzg" (UID: "4e0f2523-eb5b-4fa4-a7df-769a9c01aae5") : secret "kube-auth-proxy-tls" not found Jul 16 11:14:16.127275 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.127241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tmp\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.157098 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.157066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlnz\" (UniqueName: \"kubernetes.io/projected/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-kube-api-access-frlnz\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.225708 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.225603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-metrics-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.225708 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.225676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njch\" (UniqueName: \"kubernetes.io/projected/da6efd8b-3ef2-4346-808a-f6216d0ac660-kube-api-access-6njch\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.225941 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.225713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da6efd8b-3ef2-4346-808a-f6216d0ac660-manager-config\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.225941 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.225845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.226517 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.226491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da6efd8b-3ef2-4346-808a-f6216d0ac660-manager-config\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.228191 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.228159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-metrics-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.228729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.228708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da6efd8b-3ef2-4346-808a-f6216d0ac660-cert\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.254497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.254460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njch\" (UniqueName: \"kubernetes.io/projected/da6efd8b-3ef2-4346-808a-f6216d0ac660-kube-api-access-6njch\") pod \"lws-controller-manager-796df6bb6-8mbsw\" (UID: \"da6efd8b-3ef2-4346-808a-f6216d0ac660\") " pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.363367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.363327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:16.590923 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:14:16.590899 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6efd8b_3ef2_4346_808a_f6216d0ac660.slice/crio-72586fdba60bb9cb0b96bad5488dae2f3c8f3ba95d775ee9be76c9a8db900e6f WatchSource:0}: Error finding container 72586fdba60bb9cb0b96bad5488dae2f3c8f3ba95d775ee9be76c9a8db900e6f: Status 404 returned error can't find the container with id 72586fdba60bb9cb0b96bad5488dae2f3c8f3ba95d775ee9be76c9a8db900e6f Jul 16 11:14:16.600222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.600199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw"] Jul 16 11:14:16.630285 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.630252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.633614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.633587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f2523-eb5b-4fa4-a7df-769a9c01aae5-tls-certs\") pod \"kube-auth-proxy-79cdcbc7df-w8bzg\" (UID: \"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5\") " pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:16.841679 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:16.841592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" Jul 16 11:14:17.000051 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:17.000008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg"] Jul 16 11:14:17.005067 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:14:17.005025 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0f2523_eb5b_4fa4_a7df_769a9c01aae5.slice/crio-fabf5e3fa58085de26b7c346e6247f6a3a26257bb4468adf4280e0d3fcbe26ad WatchSource:0}: Error finding container fabf5e3fa58085de26b7c346e6247f6a3a26257bb4468adf4280e0d3fcbe26ad: Status 404 returned error can't find the container with id fabf5e3fa58085de26b7c346e6247f6a3a26257bb4468adf4280e0d3fcbe26ad Jul 16 11:14:17.282240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:17.282201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" event={"ID":"da6efd8b-3ef2-4346-808a-f6216d0ac660","Type":"ContainerStarted","Data":"72586fdba60bb9cb0b96bad5488dae2f3c8f3ba95d775ee9be76c9a8db900e6f"} Jul 16 11:14:17.283427 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:17.283398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" event={"ID":"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5","Type":"ContainerStarted","Data":"fabf5e3fa58085de26b7c346e6247f6a3a26257bb4468adf4280e0d3fcbe26ad"} Jul 16 11:14:18.289452 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:18.289414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" event={"ID":"da6efd8b-3ef2-4346-808a-f6216d0ac660","Type":"ContainerStarted","Data":"aa604840a1448b2ca136318c253e081625c28df101fd439d17ad01515170bb7d"} Jul 16 11:14:18.289822 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:18.289570 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:14:21.306184 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:21.306145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" event={"ID":"4e0f2523-eb5b-4fa4-a7df-769a9c01aae5","Type":"ContainerStarted","Data":"877e73bdcc00f0d8505224f202b95770e36988cd8f36515e407369a750a5534c"} Jul 16 11:14:21.354785 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:21.354714 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79cdcbc7df-w8bzg" podStartSLOduration=2.607889584 podStartE2EDuration="6.354697466s" totalImagesPullingTime="3.746807882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:14:21.353928771 +0000 UTC m=+345.120061909" watchObservedRunningTime="2026-07-16 11:14:21.354697466 +0000 UTC m=+345.120830603" Jul 16 11:14:21.355858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:21.355825 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" podStartSLOduration=4.731919624 podStartE2EDuration="6.355813072s" totalImagesPullingTime="1.623893448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:14:18.378955264 +0000 UTC m=+342.145088401" watchObservedRunningTime="2026-07-16 11:14:21.355813072 +0000 UTC m=+345.121946210" Jul 16 11:14:29.297256 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:14:29.297123 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-796df6bb6-8mbsw" Jul 16 11:15:21.133044 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.133013 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c7665bd7d-n5ncj"] Jul 16 11:15:21.135260 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.135244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.148822 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.148787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7665bd7d-n5ncj"] Jul 16 11:15:21.277437 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277437 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-service-ca\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-trusted-ca-bundle\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhvq\" (UniqueName: \"kubernetes.io/projected/35c33c4f-7568-4a6d-9047-00f7f37d0117-kube-api-access-qqhvq\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-oauth-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.277767 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.277646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-oauth-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378229 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhvq\" (UniqueName: \"kubernetes.io/projected/35c33c4f-7568-4a6d-9047-00f7f37d0117-kube-api-access-qqhvq\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-oauth-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-oauth-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-service-ca\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.378444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.378435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-trusted-ca-bundle\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.379271 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.379237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-oauth-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.379405 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.379244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-service-ca\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.379405 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.379292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.379405 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.379324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c33c4f-7568-4a6d-9047-00f7f37d0117-trusted-ca-bundle\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.380786 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.380753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-oauth-config\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.380889 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.380839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c33c4f-7568-4a6d-9047-00f7f37d0117-console-serving-cert\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.386648 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.386578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhvq\" (UniqueName: \"kubernetes.io/projected/35c33c4f-7568-4a6d-9047-00f7f37d0117-kube-api-access-qqhvq\") pod \"console-7c7665bd7d-n5ncj\" (UID: \"35c33c4f-7568-4a6d-9047-00f7f37d0117\") " pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.444442 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.444405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:21.571493 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:21.571349 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7665bd7d-n5ncj"] Jul 16 11:15:21.574116 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:15:21.574088 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c33c4f_7568_4a6d_9047_00f7f37d0117.slice/crio-9e634220be1a79c4d1d79c5c854301a55c0b9b3f5d0ec54dbfbb0c8aa04db390 WatchSource:0}: Error finding container 9e634220be1a79c4d1d79c5c854301a55c0b9b3f5d0ec54dbfbb0c8aa04db390: Status 404 returned error can't find the container with id 9e634220be1a79c4d1d79c5c854301a55c0b9b3f5d0ec54dbfbb0c8aa04db390 Jul 16 11:15:22.584329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:22.584285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7665bd7d-n5ncj" event={"ID":"35c33c4f-7568-4a6d-9047-00f7f37d0117","Type":"ContainerStarted","Data":"15885b7ed3fd21171b3046d931dd27aaad09c31003b37380b33d34b35b4679fe"} Jul 16 11:15:22.584799 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:22.584343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7665bd7d-n5ncj" event={"ID":"35c33c4f-7568-4a6d-9047-00f7f37d0117","Type":"ContainerStarted","Data":"9e634220be1a79c4d1d79c5c854301a55c0b9b3f5d0ec54dbfbb0c8aa04db390"} Jul 16 11:15:22.602023 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:22.601967 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c7665bd7d-n5ncj" podStartSLOduration=1.601944439 podStartE2EDuration="1.601944439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:15:22.601889988 +0000 UTC m=+406.368023135" watchObservedRunningTime="2026-07-16 11:15:22.601944439 +0000 UTC m=+406.368077576" Jul 16 11:15:31.445328 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:31.445290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:31.445769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:31.445352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:31.449882 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:31.449850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:31.627391 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:31.627345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c7665bd7d-n5ncj" Jul 16 11:15:31.681540 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:31.681508 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:15:32.274217 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.274183 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x"] Jul 16 11:15:32.277582 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.277561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.279910 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.279889 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 11:15:32.280032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.279968 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 11:15:32.280090 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.280027 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7tfn\"" Jul 16 11:15:32.295931 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.295899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x"] Jul 16 11:15:32.372453 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.372413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8rv\" (UniqueName: \"kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.372453 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.372455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.372656 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.372549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.473181 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.473139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.473683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.473220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8rv\" (UniqueName: \"kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.473683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.473252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.473683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.473557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.473683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.473631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.482275 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.482251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8rv\" (UniqueName: \"kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.587416 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.587292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:32.720707 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.720686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x"] Jul 16 11:15:32.723426 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:15:32.723402 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2643461_6c50_43f3_a2ee_46b0af0b917d.slice/crio-533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45 WatchSource:0}: Error finding container 533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45: Status 404 returned error can't find the container with id 533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45 Jul 16 11:15:32.885305 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.885274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42"] Jul 16 11:15:32.890414 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.890305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:32.898053 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:32.898026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42"] Jul 16 11:15:33.079228 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.079190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqzzd\" (UniqueName: \"kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.079440 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.079330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.079440 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.079370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.179920 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.179832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.179920 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.179875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.179920 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.179897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzzd\" (UniqueName: \"kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.180304 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.180278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.180353 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.180336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.188587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.188555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqzzd\" (UniqueName: \"kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.202945 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.202906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:33.274032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.274001 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs"] Jul 16 11:15:33.277564 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.277541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.290611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.290546 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs"] Jul 16 11:15:33.338197 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.338175 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42"] Jul 16 11:15:33.340579 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:15:33.340553 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b470038_adae_4829_a1d8_30ac699ab60b.slice/crio-b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4 WatchSource:0}: Error finding container b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4: Status 404 returned error can't find the container with id b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4 Jul 16 11:15:33.380946 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.380922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxzk\" (UniqueName: \"kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.381068 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.380964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.381068 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.380981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.481689 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.481657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxzk\" (UniqueName: \"kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.482099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.481717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.482099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.481746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.482099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.482066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.482248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.482157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.490109 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.490090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxzk\" (UniqueName: \"kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.588400 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.588341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:33.634244 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.634206 2580 generic.go:356] "Generic (PLEG): container finished" podID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerID="ed6ea5e10ae8b2601a3bc7807f7491c42897bcb4270d9fb456501499783e74c2" exitCode=0 Jul 16 11:15:33.634438 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.634285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" event={"ID":"f2643461-6c50-43f3-a2ee-46b0af0b917d","Type":"ContainerDied","Data":"ed6ea5e10ae8b2601a3bc7807f7491c42897bcb4270d9fb456501499783e74c2"} Jul 16 11:15:33.634438 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.634342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" event={"ID":"f2643461-6c50-43f3-a2ee-46b0af0b917d","Type":"ContainerStarted","Data":"533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45"} Jul 16 11:15:33.635983 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.635950 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b470038-adae-4829-a1d8-30ac699ab60b" containerID="997802baf8af93c31503d75493ed2ca5e15ecc6d6f2d8a3a14422af7cd65003b" exitCode=0 Jul 16 11:15:33.636114 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.636001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" event={"ID":"7b470038-adae-4829-a1d8-30ac699ab60b","Type":"ContainerDied","Data":"997802baf8af93c31503d75493ed2ca5e15ecc6d6f2d8a3a14422af7cd65003b"} Jul 16 11:15:33.636114 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.636035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" event={"ID":"7b470038-adae-4829-a1d8-30ac699ab60b","Type":"ContainerStarted","Data":"b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4"} Jul 16 11:15:33.680773 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.680738 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq"] Jul 16 11:15:33.686463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.686438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.697269 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.697200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq"] Jul 16 11:15:33.736525 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.736488 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs"] Jul 16 11:15:33.738656 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:15:33.738625 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7d24bd_bd96_4bd4_938f_4678b9687f2b.slice/crio-c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8 WatchSource:0}: Error finding container c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8: Status 404 returned error can't find the container with id c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8 Jul 16 11:15:33.783084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.783058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.783194 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.783103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.783283 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.783264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrnt\" (UniqueName: \"kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.883729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.883691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrnt\" (UniqueName: \"kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.883887 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.883745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.883887 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.883779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.884143 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.884123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.884179 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.884154 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.892363 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.892336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrnt\" (UniqueName: \"kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:33.996831 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:33.996757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:34.124889 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.124868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq"] Jul 16 11:15:34.126730 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:15:34.126709 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e66f54a_3237_48f4_be15_07869e2325c4.slice/crio-ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712 WatchSource:0}: Error finding container ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712: Status 404 returned error can't find the container with id ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712 Jul 16 11:15:34.642140 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.642106 2580 generic.go:356] "Generic (PLEG): container finished" podID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerID="6fc2e0e6ea8075b4f1f9cecc14288c05ddf1fe04c8ee18dcda795fe7d3ed0c08" exitCode=0 Jul 16 11:15:34.642618 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.642180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" event={"ID":"f2643461-6c50-43f3-a2ee-46b0af0b917d","Type":"ContainerDied","Data":"6fc2e0e6ea8075b4f1f9cecc14288c05ddf1fe04c8ee18dcda795fe7d3ed0c08"} Jul 16 11:15:34.643811 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.643725 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerID="93f3ea542cbfc3d84cad033418dc7c917d19ec29e79390ce47645c7b4e58ad40" exitCode=0 Jul 16 11:15:34.643880 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.643815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" event={"ID":"4c7d24bd-bd96-4bd4-938f-4678b9687f2b","Type":"ContainerDied","Data":"93f3ea542cbfc3d84cad033418dc7c917d19ec29e79390ce47645c7b4e58ad40"} Jul 16 11:15:34.643880 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.643864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" event={"ID":"4c7d24bd-bd96-4bd4-938f-4678b9687f2b","Type":"ContainerStarted","Data":"c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8"} Jul 16 11:15:34.645343 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.645323 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e66f54a-3237-48f4-be15-07869e2325c4" containerID="bfea0087a47c2a2dfc286d335bd8afc57606dfc6a1c5c0b357a33c8628269bd3" exitCode=0 Jul 16 11:15:34.645470 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.645356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" event={"ID":"1e66f54a-3237-48f4-be15-07869e2325c4","Type":"ContainerDied","Data":"bfea0087a47c2a2dfc286d335bd8afc57606dfc6a1c5c0b357a33c8628269bd3"} Jul 16 11:15:34.645470 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:34.645405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" event={"ID":"1e66f54a-3237-48f4-be15-07869e2325c4","Type":"ContainerStarted","Data":"ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712"} Jul 16 11:15:35.651635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:35.651599 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e66f54a-3237-48f4-be15-07869e2325c4" containerID="fb97e0158c8cdc71e93b08add1e2b9e3ed76efa76f3e7dd0213f86345fefdd53" exitCode=0 Jul 16 11:15:35.652071 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:35.651689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" event={"ID":"1e66f54a-3237-48f4-be15-07869e2325c4","Type":"ContainerDied","Data":"fb97e0158c8cdc71e93b08add1e2b9e3ed76efa76f3e7dd0213f86345fefdd53"} Jul 16 11:15:35.653987 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:35.653962 2580 generic.go:356] "Generic (PLEG): container finished" podID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerID="2c2a144dd6e7fcdb1668f872cb3a7631c26acbac602274d33cd2cdcdbbd49b87" exitCode=0 Jul 16 11:15:35.654077 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:35.653995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" event={"ID":"f2643461-6c50-43f3-a2ee-46b0af0b917d","Type":"ContainerDied","Data":"2c2a144dd6e7fcdb1668f872cb3a7631c26acbac602274d33cd2cdcdbbd49b87"} Jul 16 11:15:36.659708 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.659677 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b470038-adae-4829-a1d8-30ac699ab60b" containerID="05914fe5dcf7ffcbe8fc5aee1df9d983b72d829a5a812b48ecbb7ea8ad624c85" exitCode=0 Jul 16 11:15:36.660126 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.659761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" event={"ID":"7b470038-adae-4829-a1d8-30ac699ab60b","Type":"ContainerDied","Data":"05914fe5dcf7ffcbe8fc5aee1df9d983b72d829a5a812b48ecbb7ea8ad624c85"} Jul 16 11:15:36.661945 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.661927 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e66f54a-3237-48f4-be15-07869e2325c4" containerID="2fffd97efaa7553316f72da728b58cdd364a7813ae5409028aa8c5f65f519022" exitCode=0 Jul 16 11:15:36.662034 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.662011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" event={"ID":"1e66f54a-3237-48f4-be15-07869e2325c4","Type":"ContainerDied","Data":"2fffd97efaa7553316f72da728b58cdd364a7813ae5409028aa8c5f65f519022"} Jul 16 11:15:36.802463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.802437 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:36.810324 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.810300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr8rv\" (UniqueName: \"kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv\") pod \"f2643461-6c50-43f3-a2ee-46b0af0b917d\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " Jul 16 11:15:36.810473 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.810346 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle\") pod \"f2643461-6c50-43f3-a2ee-46b0af0b917d\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " Jul 16 11:15:36.810859 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.810833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle" (OuterVolumeSpecName: "bundle") pod "f2643461-6c50-43f3-a2ee-46b0af0b917d" (UID: "f2643461-6c50-43f3-a2ee-46b0af0b917d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:36.812643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.812621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv" (OuterVolumeSpecName: "kube-api-access-vr8rv") pod "f2643461-6c50-43f3-a2ee-46b0af0b917d" (UID: "f2643461-6c50-43f3-a2ee-46b0af0b917d"). InnerVolumeSpecName "kube-api-access-vr8rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:15:36.911703 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.911673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util\") pod \"f2643461-6c50-43f3-a2ee-46b0af0b917d\" (UID: \"f2643461-6c50-43f3-a2ee-46b0af0b917d\") " Jul 16 11:15:36.911916 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.911899 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8rv\" (UniqueName: \"kubernetes.io/projected/f2643461-6c50-43f3-a2ee-46b0af0b917d-kube-api-access-vr8rv\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:36.911956 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.911923 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:36.917190 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:36.917155 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util" (OuterVolumeSpecName: "util") pod "f2643461-6c50-43f3-a2ee-46b0af0b917d" (UID: "f2643461-6c50-43f3-a2ee-46b0af0b917d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:37.013293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.013261 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2643461-6c50-43f3-a2ee-46b0af0b917d-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:37.668903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.668872 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b470038-adae-4829-a1d8-30ac699ab60b" containerID="96ec479754cbb186138f9b8987a2ecb4ed90c96cd24da51c4eadbaf66d2db856" exitCode=0 Jul 16 11:15:37.669342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.668956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" event={"ID":"7b470038-adae-4829-a1d8-30ac699ab60b","Type":"ContainerDied","Data":"96ec479754cbb186138f9b8987a2ecb4ed90c96cd24da51c4eadbaf66d2db856"} Jul 16 11:15:37.670858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.670834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" event={"ID":"f2643461-6c50-43f3-a2ee-46b0af0b917d","Type":"ContainerDied","Data":"533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45"} Jul 16 11:15:37.670966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.670874 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533895b8fe520eb2c9a1b90c54e51f52b2a3e1a51c9157ff75900917a333cc45" Jul 16 11:15:37.670966 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.670838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x" Jul 16 11:15:37.672600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.672577 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerID="83066a730c71b5741d96d2991da7829b48695cf2b42889de55c332b050d89768" exitCode=0 Jul 16 11:15:37.672692 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.672661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" event={"ID":"4c7d24bd-bd96-4bd4-938f-4678b9687f2b","Type":"ContainerDied","Data":"83066a730c71b5741d96d2991da7829b48695cf2b42889de55c332b050d89768"} Jul 16 11:15:37.809942 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.809913 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:37.821233 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.821209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util\") pod \"1e66f54a-3237-48f4-be15-07869e2325c4\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " Jul 16 11:15:37.821370 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.821276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle\") pod \"1e66f54a-3237-48f4-be15-07869e2325c4\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " Jul 16 11:15:37.821370 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.821313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbrnt\" (UniqueName: \"kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt\") pod \"1e66f54a-3237-48f4-be15-07869e2325c4\" (UID: \"1e66f54a-3237-48f4-be15-07869e2325c4\") " Jul 16 11:15:37.821792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.821765 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle" (OuterVolumeSpecName: "bundle") pod "1e66f54a-3237-48f4-be15-07869e2325c4" (UID: "1e66f54a-3237-48f4-be15-07869e2325c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:37.823245 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.823217 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt" (OuterVolumeSpecName: "kube-api-access-cbrnt") pod "1e66f54a-3237-48f4-be15-07869e2325c4" (UID: "1e66f54a-3237-48f4-be15-07869e2325c4"). InnerVolumeSpecName "kube-api-access-cbrnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:15:37.827327 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.827300 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util" (OuterVolumeSpecName: "util") pod "1e66f54a-3237-48f4-be15-07869e2325c4" (UID: "1e66f54a-3237-48f4-be15-07869e2325c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:37.922188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.922153 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:37.922188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.922179 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrnt\" (UniqueName: \"kubernetes.io/projected/1e66f54a-3237-48f4-be15-07869e2325c4-kube-api-access-cbrnt\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:37.922188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:37.922188 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e66f54a-3237-48f4-be15-07869e2325c4-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:38.678432 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.678392 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" Jul 16 11:15:38.678896 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.678391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq" event={"ID":"1e66f54a-3237-48f4-be15-07869e2325c4","Type":"ContainerDied","Data":"ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712"} Jul 16 11:15:38.678896 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.678544 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6f05eebf9d016e400a3fa9902be332bf50d0ad2554215c2e0af38228ad4712" Jul 16 11:15:38.680513 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.680486 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerID="58476924d7dfbae2f6582c43e2450651394ccde02fe2c95004191203c2dcda5c" exitCode=0 Jul 16 11:15:38.680624 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.680553 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" event={"ID":"4c7d24bd-bd96-4bd4-938f-4678b9687f2b","Type":"ContainerDied","Data":"58476924d7dfbae2f6582c43e2450651394ccde02fe2c95004191203c2dcda5c"} Jul 16 11:15:38.808528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.808507 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:38.830940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.830915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle\") pod \"7b470038-adae-4829-a1d8-30ac699ab60b\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " Jul 16 11:15:38.831102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.830975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqzzd\" (UniqueName: \"kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd\") pod \"7b470038-adae-4829-a1d8-30ac699ab60b\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " Jul 16 11:15:38.831102 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.831025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util\") pod \"7b470038-adae-4829-a1d8-30ac699ab60b\" (UID: \"7b470038-adae-4829-a1d8-30ac699ab60b\") " Jul 16 11:15:38.831950 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.831910 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle" (OuterVolumeSpecName: "bundle") pod "7b470038-adae-4829-a1d8-30ac699ab60b" (UID: "7b470038-adae-4829-a1d8-30ac699ab60b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:38.833620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.833596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd" (OuterVolumeSpecName: "kube-api-access-xqzzd") pod "7b470038-adae-4829-a1d8-30ac699ab60b" (UID: "7b470038-adae-4829-a1d8-30ac699ab60b"). InnerVolumeSpecName "kube-api-access-xqzzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:15:38.837193 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.837172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util" (OuterVolumeSpecName: "util") pod "7b470038-adae-4829-a1d8-30ac699ab60b" (UID: "7b470038-adae-4829-a1d8-30ac699ab60b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:38.932884 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.932848 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:38.932884 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.932879 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b470038-adae-4829-a1d8-30ac699ab60b-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:38.932884 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:38.932888 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqzzd\" (UniqueName: \"kubernetes.io/projected/7b470038-adae-4829-a1d8-30ac699ab60b-kube-api-access-xqzzd\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:39.687427 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.687396 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" Jul 16 11:15:39.687427 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.687416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42" event={"ID":"7b470038-adae-4829-a1d8-30ac699ab60b","Type":"ContainerDied","Data":"b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4"} Jul 16 11:15:39.687834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.687462 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88cde412198d44e50a8b86e1c7a4d99e7c7ca7c3e8e206c19e519e4b48986d4" Jul 16 11:15:39.814492 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.814468 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:39.838804 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.838774 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util\") pod \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " Jul 16 11:15:39.838984 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.838826 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle\") pod \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " Jul 16 11:15:39.838984 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.838860 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmxzk\" (UniqueName: \"kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk\") pod \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\" (UID: \"4c7d24bd-bd96-4bd4-938f-4678b9687f2b\") " Jul 16 11:15:39.839714 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.839430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle" (OuterVolumeSpecName: "bundle") pod "4c7d24bd-bd96-4bd4-938f-4678b9687f2b" (UID: "4c7d24bd-bd96-4bd4-938f-4678b9687f2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:39.841047 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.841026 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk" (OuterVolumeSpecName: "kube-api-access-pmxzk") pod "4c7d24bd-bd96-4bd4-938f-4678b9687f2b" (UID: "4c7d24bd-bd96-4bd4-938f-4678b9687f2b"). InnerVolumeSpecName "kube-api-access-pmxzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:15:39.843676 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.843642 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util" (OuterVolumeSpecName: "util") pod "4c7d24bd-bd96-4bd4-938f-4678b9687f2b" (UID: "4c7d24bd-bd96-4bd4-938f-4678b9687f2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:15:39.939695 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.939606 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:39.939695 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.939640 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxzk\" (UniqueName: \"kubernetes.io/projected/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-kube-api-access-pmxzk\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:39.939695 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:39.939651 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7d24bd-bd96-4bd4-938f-4678b9687f2b-util\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:40.694286 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:40.694259 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" Jul 16 11:15:40.694696 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:40.694283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs" event={"ID":"4c7d24bd-bd96-4bd4-938f-4678b9687f2b","Type":"ContainerDied","Data":"c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8"} Jul 16 11:15:40.694696 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:40.694327 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5a9b3b2dc04e143eecef807dfaeac50c95d38f8c47b41f69a4823aa2c00a8e8" Jul 16 11:15:56.709755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:56.709689 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fb58fd9bf-mjrfp" podUID="1a050615-1920-4d81-a90d-49ecb0978934" containerName="console" containerID="cri-o://c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b" gracePeriod=15 Jul 16 11:15:56.964077 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:56.964013 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:15:57.071543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071753 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071561 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071753 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071753 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071719 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgcdx\" (UniqueName: \"kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert\") pod \"1a050615-1920-4d81-a90d-49ecb0978934\" (UID: \"1a050615-1920-4d81-a90d-49ecb0978934\") " Jul 16 11:15:57.071997 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071966 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config" (OuterVolumeSpecName: "console-config") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:15:57.072062 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.071991 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:15:57.072119 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.072072 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:15:57.072290 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.072268 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-trusted-ca-bundle\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.072359 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.072296 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-service-ca\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.072359 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.072309 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-console-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.072359 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.072321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 11:15:57.073762 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.073737 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:15:57.073903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.073883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:15:57.074020 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.073997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx" (OuterVolumeSpecName: "kube-api-access-kgcdx") pod "1a050615-1920-4d81-a90d-49ecb0978934" (UID: "1a050615-1920-4d81-a90d-49ecb0978934"). InnerVolumeSpecName "kube-api-access-kgcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:15:57.173296 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.173258 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcdx\" (UniqueName: \"kubernetes.io/projected/1a050615-1920-4d81-a90d-49ecb0978934-kube-api-access-kgcdx\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.173296 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.173290 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a050615-1920-4d81-a90d-49ecb0978934-oauth-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.173296 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.173301 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-oauth-config\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.173581 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.173311 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a050615-1920-4d81-a90d-49ecb0978934-console-serving-cert\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:15:57.775587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.775509 2580 generic.go:356] "Generic (PLEG): container finished" podID="1a050615-1920-4d81-a90d-49ecb0978934" containerID="c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b" exitCode=0 Jul 16 11:15:57.775587 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.775567 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb58fd9bf-mjrfp" Jul 16 11:15:57.776106 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.775617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb58fd9bf-mjrfp" event={"ID":"1a050615-1920-4d81-a90d-49ecb0978934","Type":"ContainerDied","Data":"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b"} Jul 16 11:15:57.776106 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.775676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb58fd9bf-mjrfp" event={"ID":"1a050615-1920-4d81-a90d-49ecb0978934","Type":"ContainerDied","Data":"836b3bea1dff4a4d4281c6fe448a72a01b42fdd4c7f910f1c1824c8afe01a17a"} Jul 16 11:15:57.776106 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.775706 2580 scope.go:117] "RemoveContainer" containerID="c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b" Jul 16 11:15:57.784858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.784839 2580 scope.go:117] "RemoveContainer" containerID="c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b" Jul 16 11:15:57.785183 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:15:57.785163 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b\": container with ID starting with c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b not found: ID does not exist" containerID="c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b" Jul 16 11:15:57.785231 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.785192 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b"} err="failed to get container status \"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b\": rpc error: code = NotFound desc = could not find container \"c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b\": container with ID starting with c4de22d935fdd0a7e90fbbe0ce8e456bc5661aaf890696deb6a593ecb9ae105b not found: ID does not exist" Jul 16 11:15:57.793495 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:15:57.793458 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a050615_1920_4d81_a90d_49ecb0978934.slice\": RecentStats: unable to find data in memory cache]" Jul 16 11:15:57.804496 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.804455 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:15:57.807100 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:57.807074 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fb58fd9bf-mjrfp"] Jul 16 11:15:58.802115 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:15:58.802083 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a050615-1920-4d81-a90d-49ecb0978934" path="/var/lib/kubelet/pods/1a050615-1920-4d81-a90d-49ecb0978934/volumes" Jul 16 11:16:00.562013 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.561983 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9"] Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562293 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="pull" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="pull" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562315 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562322 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562329 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a050615-1920-4d81-a90d-49ecb0978934" containerName="console" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562335 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a050615-1920-4d81-a90d-49ecb0978934" containerName="console" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562344 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="pull" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562349 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="pull" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562355 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562360 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562368 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562386 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="extract" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562395 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="util" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562400 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="util" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562408 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="util" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562413 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="util" Jul 16 11:16:00.562413 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562420 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="pull" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562426 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="pull" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562431 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562436 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562442 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="pull" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562447 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="pull" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562453 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="util" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562458 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="util" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562463 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="util" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562469 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="util" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562516 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2643461-6c50-43f3-a2ee-46b0af0b917d" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562523 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b470038-adae-4829-a1d8-30ac699ab60b" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562531 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e66f54a-3237-48f4-be15-07869e2325c4" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562537 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7d24bd-bd96-4bd4-938f-4678b9687f2b" containerName="extract" Jul 16 11:16:00.562903 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.562541 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a050615-1920-4d81-a90d-49ecb0978934" containerName="console" Jul 16 11:16:00.565488 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.565469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.568332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.568311 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 11:16:00.568459 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.568311 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 11:16:00.568459 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.568402 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 11:16:00.569149 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.569129 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 11:16:00.569233 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.569157 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7tfn\"" Jul 16 11:16:00.575874 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.575856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9"] Jul 16 11:16:00.602984 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.602956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.603141 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.602992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjv7\" (UniqueName: \"kubernetes.io/projected/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-kube-api-access-pjjv7\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.603141 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.603017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.704154 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.704116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.704154 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.704160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjv7\" (UniqueName: \"kubernetes.io/projected/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-kube-api-access-pjjv7\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.704426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.704186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.704426 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:16:00.704282 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 16 11:16:00.704426 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:16:00.704367 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert podName:1e8a7b3c-42f3-4971-946c-c42bf2eaff4e nodeName:}" failed. No retries permitted until 2026-07-16 11:16:01.204346019 +0000 UTC m=+444.970479150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-m7bz9" (UID: "1e8a7b3c-42f3-4971-946c-c42bf2eaff4e") : secret "plugin-serving-cert" not found Jul 16 11:16:00.704848 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.704829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:00.713547 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:00.713523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjv7\" (UniqueName: \"kubernetes.io/projected/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-kube-api-access-pjjv7\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:01.208635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:01.208595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:01.210982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:01.210957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8a7b3c-42f3-4971-946c-c42bf2eaff4e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-m7bz9\" (UID: \"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:01.475528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:01.475424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" Jul 16 11:16:01.605151 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:01.605082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9"] Jul 16 11:16:01.607423 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:16:01.607396 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e8a7b3c_42f3_4971_946c_c42bf2eaff4e.slice/crio-ba951b699a072b6687597d7b9beb480ecb9a709cc33da8c50c09ec9f6bae6e57 WatchSource:0}: Error finding container ba951b699a072b6687597d7b9beb480ecb9a709cc33da8c50c09ec9f6bae6e57: Status 404 returned error can't find the container with id ba951b699a072b6687597d7b9beb480ecb9a709cc33da8c50c09ec9f6bae6e57 Jul 16 11:16:01.796246 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:01.796162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" event={"ID":"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e","Type":"ContainerStarted","Data":"ba951b699a072b6687597d7b9beb480ecb9a709cc33da8c50c09ec9f6bae6e57"} Jul 16 11:16:24.923479 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:24.923440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" event={"ID":"1e8a7b3c-42f3-4971-946c-c42bf2eaff4e","Type":"ContainerStarted","Data":"f74854ce2258be8f5a0c27f5002c1a1b9ff4f3850d4b2df101ca5b575efcefa1"} Jul 16 11:16:24.940488 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:24.940372 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-m7bz9" podStartSLOduration=2.135959681 podStartE2EDuration="24.940356859s" totalImagesPullingTime="22.804397178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:16:24.939556272 +0000 UTC m=+468.705689408" watchObservedRunningTime="2026-07-16 11:16:24.940356859 +0000 UTC m=+468.706489994" Jul 16 11:16:47.094899 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.094861 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:16:47.276974 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.276941 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:16:47.276974 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.276980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:16:47.277180 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.277065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.279455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.279434 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 11:16:47.409309 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.409275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2h84\" (UniqueName: \"kubernetes.io/projected/69bc7662-e7fd-489b-9853-8f2b91162b81-kube-api-access-c2h84\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.409502 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.409360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69bc7662-e7fd-489b-9853-8f2b91162b81-config-file\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.510235 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.510202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2h84\" (UniqueName: \"kubernetes.io/projected/69bc7662-e7fd-489b-9853-8f2b91162b81-kube-api-access-c2h84\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.510422 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.510294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69bc7662-e7fd-489b-9853-8f2b91162b81-config-file\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.510959 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.510942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69bc7662-e7fd-489b-9853-8f2b91162b81-config-file\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.518794 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.518767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2h84\" (UniqueName: \"kubernetes.io/projected/69bc7662-e7fd-489b-9853-8f2b91162b81-kube-api-access-c2h84\") pod \"limitador-limitador-54d6d988d5-l7v5k\" (UID: \"69bc7662-e7fd-489b-9853-8f2b91162b81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.543336 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.543304 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:47.569722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.569690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:47.569878 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.569784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:47.573021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.573002 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2rgzz\"" Jul 16 11:16:47.586488 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.586466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:47.611527 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.611439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jfb\" (UniqueName: \"kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb\") pod \"authorino-58cd955675-sm58l\" (UID: \"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4\") " pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:47.712347 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.712312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jfb\" (UniqueName: \"kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb\") pod \"authorino-58cd955675-sm58l\" (UID: \"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4\") " pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:47.721819 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.721788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jfb\" (UniqueName: \"kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb\") pod \"authorino-58cd955675-sm58l\" (UID: \"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4\") " pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:47.740929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.740906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:16:47.742676 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:16:47.742650 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bc7662_e7fd_489b_9853_8f2b91162b81.slice/crio-7ab69f4bf810ac013dbff75ca71763d5bf914ca002e4aab70d66b5e2904c0b9f WatchSource:0}: Error finding container 7ab69f4bf810ac013dbff75ca71763d5bf914ca002e4aab70d66b5e2904c0b9f: Status 404 returned error can't find the container with id 7ab69f4bf810ac013dbff75ca71763d5bf914ca002e4aab70d66b5e2904c0b9f Jul 16 11:16:47.879247 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:47.879206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:48.042032 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:48.042007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:48.042428 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:48.042406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" event={"ID":"69bc7662-e7fd-489b-9853-8f2b91162b81","Type":"ContainerStarted","Data":"7ab69f4bf810ac013dbff75ca71763d5bf914ca002e4aab70d66b5e2904c0b9f"} Jul 16 11:16:48.043890 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:16:48.043866 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca21cb0_6a52_4b4b_acfe_1b5e5c353cd4.slice/crio-3c6432543eba0002cd182e24ef1e80b2333b3b04c19dfb2039d4432071897c1e WatchSource:0}: Error finding container 3c6432543eba0002cd182e24ef1e80b2333b3b04c19dfb2039d4432071897c1e: Status 404 returned error can't find the container with id 3c6432543eba0002cd182e24ef1e80b2333b3b04c19dfb2039d4432071897c1e Jul 16 11:16:49.052282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:49.052241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sm58l" event={"ID":"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4","Type":"ContainerStarted","Data":"3c6432543eba0002cd182e24ef1e80b2333b3b04c19dfb2039d4432071897c1e"} Jul 16 11:16:52.873759 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:52.873723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:55.086919 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.086881 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-sm58l" podUID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" containerName="authorino" containerID="cri-o://f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76" gracePeriod=30 Jul 16 11:16:55.086919 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.086900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sm58l" event={"ID":"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4","Type":"ContainerStarted","Data":"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76"} Jul 16 11:16:55.088682 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.088647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" event={"ID":"69bc7662-e7fd-489b-9853-8f2b91162b81","Type":"ContainerStarted","Data":"486f77d2132f93dc2542f669074f3ef8a1adfe4d26bbdd1dc3d5b877d327de45"} Jul 16 11:16:55.088825 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.088790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:16:55.104477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.104360 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-sm58l" podStartSLOduration=1.702626231 podStartE2EDuration="8.104345706s" totalImagesPullingTime="6.401719475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:16:55.103461498 +0000 UTC m=+498.869594626" watchObservedRunningTime="2026-07-16 11:16:55.104345706 +0000 UTC m=+498.870478821" Jul 16 11:16:55.120031 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.119982 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" podStartSLOduration=1.410410135 podStartE2EDuration="8.119964682s" totalImagesPullingTime="6.709554547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:16:55.118871752 +0000 UTC m=+498.885004888" watchObservedRunningTime="2026-07-16 11:16:55.119964682 +0000 UTC m=+498.886097819" Jul 16 11:16:55.830588 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.830564 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:55.883596 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.883555 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9jfb\" (UniqueName: \"kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb\") pod \"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4\" (UID: \"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4\") " Jul 16 11:16:55.885666 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.885642 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb" (OuterVolumeSpecName: "kube-api-access-d9jfb") pod "aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" (UID: "aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4"). InnerVolumeSpecName "kube-api-access-d9jfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:16:55.984833 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:55.984737 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jfb\" (UniqueName: \"kubernetes.io/projected/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4-kube-api-access-d9jfb\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:16:56.094870 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.094832 2580 generic.go:356] "Generic (PLEG): container finished" podID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" containerID="f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76" exitCode=0 Jul 16 11:16:56.094870 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.094872 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sm58l" Jul 16 11:16:56.095340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.094920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sm58l" event={"ID":"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4","Type":"ContainerDied","Data":"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76"} Jul 16 11:16:56.095340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.094960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sm58l" event={"ID":"aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4","Type":"ContainerDied","Data":"3c6432543eba0002cd182e24ef1e80b2333b3b04c19dfb2039d4432071897c1e"} Jul 16 11:16:56.095340 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.094982 2580 scope.go:117] "RemoveContainer" containerID="f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76" Jul 16 11:16:56.104096 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.104077 2580 scope.go:117] "RemoveContainer" containerID="f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76" Jul 16 11:16:56.104352 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:16:56.104333 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76\": container with ID starting with f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76 not found: ID does not exist" containerID="f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76" Jul 16 11:16:56.104431 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.104361 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76"} err="failed to get container status \"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76\": rpc error: code = NotFound desc = could not find container \"f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76\": container with ID starting with f04755e98a135f34fc9d7fc945c7b6f98a2401ec45aaef76a35a295e524b0c76 not found: ID does not exist" Jul 16 11:16:56.117123 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.117094 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:56.122756 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.122729 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-sm58l"] Jul 16 11:16:56.802111 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:16:56.802076 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" path="/var/lib/kubelet/pods/aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4/volumes" Jul 16 11:17:06.097877 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:06.097841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l7v5k" Jul 16 11:17:25.802445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.799850 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:25.802445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.800699 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" containerName="authorino" Jul 16 11:17:25.802445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.800719 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" containerName="authorino" Jul 16 11:17:25.802445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.800876 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca21cb0-6a52-4b4b-acfe-1b5e5c353cd4" containerName="authorino" Jul 16 11:17:25.810610 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.810583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:25.810759 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.810686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:25.812946 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.812925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2rgzz\"" Jul 16 11:17:25.833919 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.833896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndt7\" (UniqueName: \"kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7\") pod \"authorino-69799fbb9d-klk22\" (UID: \"1fc0faaf-15e4-45e2-ac87-a19d93652c90\") " pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:25.934686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.934649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndt7\" (UniqueName: \"kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7\") pod \"authorino-69799fbb9d-klk22\" (UID: \"1fc0faaf-15e4-45e2-ac87-a19d93652c90\") " pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:25.942173 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:25.942145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndt7\" (UniqueName: \"kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7\") pod \"authorino-69799fbb9d-klk22\" (UID: \"1fc0faaf-15e4-45e2-ac87-a19d93652c90\") " pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:26.121412 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:26.121359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:26.254737 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:26.254713 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:26.257137 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:17:26.257102 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fc0faaf_15e4_45e2_ac87_a19d93652c90.slice/crio-91ef410d2a985b2ef7e3744239592d5ea4c3cc74bc7bd0a23575ca1d32710f45 WatchSource:0}: Error finding container 91ef410d2a985b2ef7e3744239592d5ea4c3cc74bc7bd0a23575ca1d32710f45: Status 404 returned error can't find the container with id 91ef410d2a985b2ef7e3744239592d5ea4c3cc74bc7bd0a23575ca1d32710f45 Jul 16 11:17:26.393060 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:26.392975 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:27.253517 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.253480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69799fbb9d-klk22" event={"ID":"1fc0faaf-15e4-45e2-ac87-a19d93652c90","Type":"ContainerStarted","Data":"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b"} Jul 16 11:17:27.253517 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.253520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69799fbb9d-klk22" event={"ID":"1fc0faaf-15e4-45e2-ac87-a19d93652c90","Type":"ContainerStarted","Data":"91ef410d2a985b2ef7e3744239592d5ea4c3cc74bc7bd0a23575ca1d32710f45"} Jul 16 11:17:27.254072 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.253566 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-69799fbb9d-klk22" podUID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" containerName="authorino" containerID="cri-o://d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b" gracePeriod=30 Jul 16 11:17:27.270844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.270800 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69799fbb9d-klk22" podStartSLOduration=1.835910392 podStartE2EDuration="2.270783669s" totalImagesPullingTime="434.873277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:17:27.268882074 +0000 UTC m=+531.035015211" watchObservedRunningTime="2026-07-16 11:17:27.270783669 +0000 UTC m=+531.036916976" Jul 16 11:17:27.502481 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.502421 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:27.545496 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.545396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qndt7\" (UniqueName: \"kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7\") pod \"1fc0faaf-15e4-45e2-ac87-a19d93652c90\" (UID: \"1fc0faaf-15e4-45e2-ac87-a19d93652c90\") " Jul 16 11:17:27.547478 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.547453 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7" (OuterVolumeSpecName: "kube-api-access-qndt7") pod "1fc0faaf-15e4-45e2-ac87-a19d93652c90" (UID: "1fc0faaf-15e4-45e2-ac87-a19d93652c90"). InnerVolumeSpecName "kube-api-access-qndt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:17:27.646364 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:27.646328 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndt7\" (UniqueName: \"kubernetes.io/projected/1fc0faaf-15e4-45e2-ac87-a19d93652c90-kube-api-access-qndt7\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:17:28.259363 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.259327 2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" containerID="d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b" exitCode=0 Jul 16 11:17:28.259805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.259372 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69799fbb9d-klk22" Jul 16 11:17:28.259805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.259412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69799fbb9d-klk22" event={"ID":"1fc0faaf-15e4-45e2-ac87-a19d93652c90","Type":"ContainerDied","Data":"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b"} Jul 16 11:17:28.259805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.259469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69799fbb9d-klk22" event={"ID":"1fc0faaf-15e4-45e2-ac87-a19d93652c90","Type":"ContainerDied","Data":"91ef410d2a985b2ef7e3744239592d5ea4c3cc74bc7bd0a23575ca1d32710f45"} Jul 16 11:17:28.259805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.259489 2580 scope.go:117] "RemoveContainer" containerID="d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b" Jul 16 11:17:28.267834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.267815 2580 scope.go:117] "RemoveContainer" containerID="d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b" Jul 16 11:17:28.268064 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:17:28.268044 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b\": container with ID starting with d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b not found: ID does not exist" containerID="d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b" Jul 16 11:17:28.268105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.268072 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b"} err="failed to get container status \"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b\": rpc error: code = NotFound desc = could not find container \"d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b\": container with ID starting with d7e9d3a4e22a74b72cd4374136c04bbca6af5127896aa9b59c18d0c6cdd80b5b not found: ID does not exist" Jul 16 11:17:28.281169 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.281143 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:28.286115 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.286094 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69799fbb9d-klk22"] Jul 16 11:17:28.802172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:17:28.802138 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" path="/var/lib/kubelet/pods/1fc0faaf-15e4-45e2-ac87-a19d93652c90/volumes" Jul 16 11:18:09.337351 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:18:09.337310 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:18:32.430255 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:18:32.430217 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:18:36.662278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:18:36.662242 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:18:36.662749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:18:36.662570 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:18:52.733453 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:18:52.733421 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:19:06.729494 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:06.729415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:19:12.211639 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.211601 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d"] Jul 16 11:19:12.212040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.211986 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" containerName="authorino" Jul 16 11:19:12.212040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.212002 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" containerName="authorino" Jul 16 11:19:12.212127 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.212087 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc0faaf-15e4-45e2-ac87-a19d93652c90" containerName="authorino" Jul 16 11:19:12.214989 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.214975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.217934 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.217912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7wggx\"" Jul 16 11:19:12.218069 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.217962 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 11:19:12.218069 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.217978 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 11:19:12.218069 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.217990 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 11:19:12.223576 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.223707 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmxp\" (UniqueName: \"kubernetes.io/projected/5e21a10d-4961-4dff-bb7e-150e21305a16-kube-api-access-flmxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.223707 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.223809 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.223809 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e21a10d-4961-4dff-bb7e-150e21305a16-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.223809 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.223767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.226115 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.226092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d"] Jul 16 11:19:12.324330 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e21a10d-4961-4dff-bb7e-150e21305a16-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324330 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324605 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324605 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmxp\" (UniqueName: \"kubernetes.io/projected/5e21a10d-4961-4dff-bb7e-150e21305a16-kube-api-access-flmxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324605 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324605 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.324889 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.325013 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.325013 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.324974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.326682 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.326659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e21a10d-4961-4dff-bb7e-150e21305a16-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.327011 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.326992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e21a10d-4961-4dff-bb7e-150e21305a16-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.333437 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.333397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmxp\" (UniqueName: \"kubernetes.io/projected/5e21a10d-4961-4dff-bb7e-150e21305a16-kube-api-access-flmxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d\" (UID: \"5e21a10d-4961-4dff-bb7e-150e21305a16\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.526082 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.525983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:12.668560 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.668518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d"] Jul 16 11:19:12.671520 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:19:12.671482 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e21a10d_4961_4dff_bb7e_150e21305a16.slice/crio-3d21b6ce8505d23fdea5e756a836d5bb4572a0aa61d629527241e8c4102cbf43 WatchSource:0}: Error finding container 3d21b6ce8505d23fdea5e756a836d5bb4572a0aa61d629527241e8c4102cbf43: Status 404 returned error can't find the container with id 3d21b6ce8505d23fdea5e756a836d5bb4572a0aa61d629527241e8c4102cbf43 Jul 16 11:19:12.673329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.673306 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:19:12.752888 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:12.752849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" event={"ID":"5e21a10d-4961-4dff-bb7e-150e21305a16","Type":"ContainerStarted","Data":"3d21b6ce8505d23fdea5e756a836d5bb4572a0aa61d629527241e8c4102cbf43"} Jul 16 11:19:13.836997 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:13.836955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:19:14.764145 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:14.764109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" event={"ID":"5e21a10d-4961-4dff-bb7e-150e21305a16","Type":"ContainerStarted","Data":"2272acab1e532c39be08c265d45b3bad186904ce264b0b3f35f0f40dfc205ff0"} Jul 16 11:19:14.784924 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:14.784875 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" podStartSLOduration=1.752377487 podStartE2EDuration="2.784858451s" totalImagesPullingTime="1.032480964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:19:14.782781777 +0000 UTC m=+638.548914914" watchObservedRunningTime="2026-07-16 11:19:14.784858451 +0000 UTC m=+638.550991587" Jul 16 11:19:22.526548 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:22.526502 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:22.526548 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:22.526567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:22.538881 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:22.538846 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:22.811543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:22.811462 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d" Jul 16 11:19:22.834650 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:22.834603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:19:36.123171 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.123134 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5"] Jul 16 11:19:36.126670 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.126649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.128756 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.128738 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 11:19:36.137486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.137462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5"] Jul 16 11:19:36.239558 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.239558 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9916b08-432a-455d-8acd-5b4140719ee5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.239769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.239769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.239769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qfw\" (UniqueName: \"kubernetes.io/projected/d9916b08-432a-455d-8acd-5b4140719ee5-kube-api-access-z2qfw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.239769 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.239751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340313 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340313 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9916b08-432a-455d-8acd-5b4140719ee5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qfw\" (UniqueName: \"kubernetes.io/projected/d9916b08-432a-455d-8acd-5b4140719ee5-kube-api-access-z2qfw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340762 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.340983 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.340875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.342775 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.342748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9916b08-432a-455d-8acd-5b4140719ee5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.342940 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.342922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9916b08-432a-455d-8acd-5b4140719ee5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.348749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.348726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qfw\" (UniqueName: \"kubernetes.io/projected/d9916b08-432a-455d-8acd-5b4140719ee5-kube-api-access-z2qfw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5\" (UID: \"d9916b08-432a-455d-8acd-5b4140719ee5\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.437185 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.437081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:36.571620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.571594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5"] Jul 16 11:19:36.573886 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:19:36.573852 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9916b08_432a_455d_8acd_5b4140719ee5.slice/crio-5b09280212ce961a226aae00518cb5cb71fb72f04031cdfd1b59f2fc8ce7a271 WatchSource:0}: Error finding container 5b09280212ce961a226aae00518cb5cb71fb72f04031cdfd1b59f2fc8ce7a271: Status 404 returned error can't find the container with id 5b09280212ce961a226aae00518cb5cb71fb72f04031cdfd1b59f2fc8ce7a271 Jul 16 11:19:36.865108 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.865078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" event={"ID":"d9916b08-432a-455d-8acd-5b4140719ee5","Type":"ContainerStarted","Data":"777be138a2e728db36e0cc0042cafda0fcf7f0f4009d2e825c96d86ffad57dea"} Jul 16 11:19:36.865267 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.865118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" event={"ID":"d9916b08-432a-455d-8acd-5b4140719ee5","Type":"ContainerStarted","Data":"5b09280212ce961a226aae00518cb5cb71fb72f04031cdfd1b59f2fc8ce7a271"} Jul 16 11:19:36.889984 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:36.889928 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" podStartSLOduration=0.678599268 podStartE2EDuration="889.910171ms" totalImagesPullingTime="211.310903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:19:36.887927344 +0000 UTC m=+660.654060482" watchObservedRunningTime="2026-07-16 11:19:36.889910171 +0000 UTC m=+660.656043308" Jul 16 11:19:38.641210 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:38.641171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:19:46.438138 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:46.438092 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:46.438138 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:46.438140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:46.450882 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:46.450855 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:19:46.927943 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:19:46.927907 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5" Jul 16 11:21:38.806248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.806212 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9cc598755-jksjs"] Jul 16 11:21:38.809289 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.809271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.812471 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.812449 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2rgzz\"" Jul 16 11:21:38.812602 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.812513 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 11:21:38.826521 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.826493 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9cc598755-jksjs"] Jul 16 11:21:38.877073 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.877040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88136d32-8887-433c-bf8e-9a768adb9386-tls-cert\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.877232 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.877098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vrq\" (UniqueName: \"kubernetes.io/projected/88136d32-8887-433c-bf8e-9a768adb9386-kube-api-access-x8vrq\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.978230 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.978170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vrq\" (UniqueName: \"kubernetes.io/projected/88136d32-8887-433c-bf8e-9a768adb9386-kube-api-access-x8vrq\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.978435 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.978277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88136d32-8887-433c-bf8e-9a768adb9386-tls-cert\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.980738 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.980714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88136d32-8887-433c-bf8e-9a768adb9386-tls-cert\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:38.986061 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:38.986030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vrq\" (UniqueName: \"kubernetes.io/projected/88136d32-8887-433c-bf8e-9a768adb9386-kube-api-access-x8vrq\") pod \"authorino-9cc598755-jksjs\" (UID: \"88136d32-8887-433c-bf8e-9a768adb9386\") " pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:39.118550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:39.118517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9cc598755-jksjs" Jul 16 11:21:39.253219 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:39.253198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9cc598755-jksjs"] Jul 16 11:21:39.255660 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:21:39.255635 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88136d32_8887_433c_bf8e_9a768adb9386.slice/crio-1ccadc189d9360218b7602f9aacd765024598bf624730f11b06f93cac556918f WatchSource:0}: Error finding container 1ccadc189d9360218b7602f9aacd765024598bf624730f11b06f93cac556918f: Status 404 returned error can't find the container with id 1ccadc189d9360218b7602f9aacd765024598bf624730f11b06f93cac556918f Jul 16 11:21:39.457202 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:39.457119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9cc598755-jksjs" event={"ID":"88136d32-8887-433c-bf8e-9a768adb9386","Type":"ContainerStarted","Data":"1ccadc189d9360218b7602f9aacd765024598bf624730f11b06f93cac556918f"} Jul 16 11:21:40.464074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:40.464029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9cc598755-jksjs" event={"ID":"88136d32-8887-433c-bf8e-9a768adb9386","Type":"ContainerStarted","Data":"ee9d9c91dbf4093588ea159dc6b913136b118dee08cf63405af9aba52c65b09a"} Jul 16 11:21:40.494428 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:21:40.494355 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9cc598755-jksjs" podStartSLOduration=2.052145926 podStartE2EDuration="2.494336037s" totalImagesPullingTime="442.190111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:21:40.492073659 +0000 UTC m=+784.258206808" watchObservedRunningTime="2026-07-16 11:21:40.494336037 +0000 UTC m=+784.260469180" Jul 16 11:22:16.343205 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:22:16.343114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:22:27.037360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:22:27.037320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:22:37.234189 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:22:37.234148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:22:47.448656 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:22:47.448619 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:22:57.241242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:22:57.241204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:23:24.934556 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:23:24.934525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:23:35.934129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:23:35.934050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:23:36.692740 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:23:36.692712 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:23:36.695241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:23:36.695221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:24:10.334788 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:10.334751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:24:22.933635 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:22.933598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:24:30.446078 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:30.446041 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:24:37.131162 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:37.131101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:24:47.834568 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:47.834525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:24:56.336346 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:24:56.336310 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:07.938887 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:07.938796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:15.143296 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:15.143261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:22.460629 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:22.460588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:30.532946 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:30.532909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:43.341497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:43.341456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:25:52.337188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:25:52.337155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:26:16.637036 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:26:16.637002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:26:26.441183 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:26:26.441147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:26:42.944148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:26:42.944070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:26:52.733222 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:26:52.733181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:27:03.639658 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:27:03.639622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:27:12.433730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:27:12.433696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:27:21.130531 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:27:21.130494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:28:36.731394 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:28:36.731350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:28:36.733957 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:28:36.733929 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:29:10.642881 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:29:10.642843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:29:22.036799 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:29:22.036762 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:29:32.927528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:29:32.927448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:29:41.751638 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:29:41.751579 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:29:52.532368 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:29:52.532332 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:00.336389 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:00.336348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:13.536329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:13.536288 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:22.238306 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:22.238272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:31.029477 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:31.029443 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:41.358141 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:41.358108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:30:49.752150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:30:49.752109 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:31:01.338560 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:31:01.338530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:31:23.333394 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:31:23.333353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:31:32.738802 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:31:32.738765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:31:41.746773 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:31:41.746735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:31:53.735823 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:31:53.735778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:08.637823 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:08.637783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:17.826850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:17.826811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:26.575040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:26.575004 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:36.727463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:36.727369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:46.028990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:46.028954 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:32:55.040333 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:32:55.040301 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:07.930364 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:07.930327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:17.042625 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:17.042591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:26.235197 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:26.235158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:36.762080 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:36.762052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:33:36.765585 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:36.765562 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:33:38.327842 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:38.327806 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:47.138106 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:47.138070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:33:53.936066 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:33:53.936031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:34:10.746276 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:34:10.745513 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:34:20.733342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:34:20.733302 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:34:33.339528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:34:33.339486 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:34:48.926752 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:34:48.926704 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:34:56.738936 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:34:56.738899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:35:07.831483 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:35:07.831448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:35:17.831145 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:35:17.831103 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:35:27.332476 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:35:27.332436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:35:37.035346 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:35:37.035307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:37:03.337444 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:37:03.337347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:37:11.134474 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:37:11.134436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:37:19.736141 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:37:19.736108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:37:29.425166 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:37:29.425128 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:37:40.533390 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:37:40.533344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:38:07.233038 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:38:07.232993 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:38:18.630287 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:38:18.630250 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:38:36.799637 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:38:36.799596 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:38:36.804175 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:38:36.804153 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:39:09.742973 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:09.742929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:39:20.445360 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:20.445317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:39:29.839464 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.839433 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:39:29.842581 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.842565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.844756 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.844736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a0660d\"/\"default-dockercfg-96czj\"" Jul 16 11:39:29.845626 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.845606 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a0660d\"/\"kube-root-ca.crt\"" Jul 16 11:39:29.845693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.845622 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a0660d\"/\"test-model-a0660d-kserve-self-signed-certs\"" Jul 16 11:39:29.845693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.845640 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a0660d\"/\"openshift-service-ca.crt\"" Jul 16 11:39:29.854188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.854159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:39:29.897057 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.897242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.897242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.897242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjptm\" (UniqueName: \"kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.897242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.897242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.897212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998363 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998573 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998573 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjptm\" (UniqueName: \"kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998573 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998684 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998684 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998894 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.998954 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.998921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:29.999016 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:29.999001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:30.000686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.000664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:30.000942 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.000924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:30.006490 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.006465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjptm\" (UniqueName: \"kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm\") pod \"test-model-a0660d-kserve-57d748d99f-kqx6w\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:30.152267 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.152222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:30.290626 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.290596 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:39:30.292559 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:39:30.292530 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d4a4be9_9f2a_403d_bd73_07845ae9eded.slice/crio-753b26b9fb3d54289d4faa522b0f7917c967378a56f958777a10a17dd949bfe3 WatchSource:0}: Error finding container 753b26b9fb3d54289d4faa522b0f7917c967378a56f958777a10a17dd949bfe3: Status 404 returned error can't find the container with id 753b26b9fb3d54289d4faa522b0f7917c967378a56f958777a10a17dd949bfe3 Jul 16 11:39:30.735367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.735326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" event={"ID":"7d4a4be9-9f2a-403d-bd73-07845ae9eded","Type":"ContainerStarted","Data":"0cd917aae653d33bcf404caa8435d3644e206513ce18459792be68544b05035f"} Jul 16 11:39:30.735575 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.735399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" event={"ID":"7d4a4be9-9f2a-403d-bd73-07845ae9eded","Type":"ContainerStarted","Data":"753b26b9fb3d54289d4faa522b0f7917c967378a56f958777a10a17dd949bfe3"} Jul 16 11:39:30.752915 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:30.752864 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" podStartSLOduration=1.752845324 podStartE2EDuration="1.752845324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:39:30.750784059 +0000 UTC m=+1854.516917197" watchObservedRunningTime="2026-07-16 11:39:30.752845324 +0000 UTC m=+1854.518978462" Jul 16 11:39:33.426900 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:33.426865 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:39:40.152999 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:40.152960 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:40.153422 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:40.153120 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:40.165820 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:40.165791 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:40.801821 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:40.801795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:39:56.611333 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.611294 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:39:56.614736 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.614710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.618463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.618431 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca07b2\"/\"openshift-service-ca.crt\"" Jul 16 11:39:56.618463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.618448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca07b2\"/\"kube-root-ca.crt\"" Jul 16 11:39:56.618463 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.618458 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca07b2\"/\"default-dockercfg-ffnrm\"" Jul 16 11:39:56.618710 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.618473 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca07b2\"/\"test-model-ca07b2-kserve-self-signed-certs\"" Jul 16 11:39:56.627611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.627581 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:39:56.738155 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.738332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x489n\" (UniqueName: \"kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.738332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.738332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.738332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.738332 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.738321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x489n\" (UniqueName: \"kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839791 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.839861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.839827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.841698 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.841681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.841963 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.841947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.847632 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.847600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x489n\" (UniqueName: \"kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n\") pod \"test-model-ca07b2-kserve-5f9586f9ff-7blwf\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:56.924640 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:56.924537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:39:57.070621 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:57.070548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:39:57.072949 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:39:57.072918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2a609c7_b0df_432a_90ff_541709a8b8ce.slice/crio-78137a0262195eae89175180f55cf397c55c96c6641e08d45cc6a8884d7af594 WatchSource:0}: Error finding container 78137a0262195eae89175180f55cf397c55c96c6641e08d45cc6a8884d7af594: Status 404 returned error can't find the container with id 78137a0262195eae89175180f55cf397c55c96c6641e08d45cc6a8884d7af594 Jul 16 11:39:57.868283 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:57.868200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" event={"ID":"b2a609c7-b0df-432a-90ff-541709a8b8ce","Type":"ContainerStarted","Data":"05f7177185d556f8a8c1f6dcc454252bafea04ffc7aa463706e56b389c697fa0"} Jul 16 11:39:57.868283 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:57.868247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" event={"ID":"b2a609c7-b0df-432a-90ff-541709a8b8ce","Type":"ContainerStarted","Data":"78137a0262195eae89175180f55cf397c55c96c6641e08d45cc6a8884d7af594"} Jul 16 11:39:57.887106 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:39:57.887050 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" podStartSLOduration=1.887033688 podStartE2EDuration="1.887033688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:39:57.886599288 +0000 UTC m=+1881.652732437" watchObservedRunningTime="2026-07-16 11:39:57.887033688 +0000 UTC m=+1881.653166802" Jul 16 11:40:00.630922 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:00.630887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:40:06.924861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:06.924817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:06.924861 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:06.924874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:06.942881 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:06.942854 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:07.930904 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:07.930877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:13.833550 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:13.833515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:40:17.893442 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:17.893407 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:40:18.817308 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:18.817273 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:40:31.238347 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:31.238304 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:40:32.913819 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:32.913759 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" podUID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" containerName="main" containerID="cri-o://0cd917aae653d33bcf404caa8435d3644e206513ce18459792be68544b05035f" gracePeriod=45 Jul 16 11:40:33.042196 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.042166 2580 generic.go:356] "Generic (PLEG): container finished" podID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" containerID="0cd917aae653d33bcf404caa8435d3644e206513ce18459792be68544b05035f" exitCode=0 Jul 16 11:40:33.042344 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.042246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" event={"ID":"7d4a4be9-9f2a-403d-bd73-07845ae9eded","Type":"ContainerDied","Data":"0cd917aae653d33bcf404caa8435d3644e206513ce18459792be68544b05035f"} Jul 16 11:40:33.170579 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.170503 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:40:33.263402 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263430 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263490 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263510 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263600 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjptm\" (UniqueName: \"kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm\") pod \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\" (UID: \"7d4a4be9-9f2a-403d-bd73-07845ae9eded\") " Jul 16 11:40:33.263842 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home" (OuterVolumeSpecName: "home") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:33.263842 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263784 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:33.263842 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:33.264004 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263893 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.264004 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.263908 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.265687 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.265666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:40:33.265755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.265678 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm" (OuterVolumeSpecName: "kube-api-access-sjptm") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "kube-api-access-sjptm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:40:33.265755 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.265733 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm" (OuterVolumeSpecName: "dshm") pod "7d4a4be9-9f2a-403d-bd73-07845ae9eded" (UID: "7d4a4be9-9f2a-403d-bd73-07845ae9eded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:33.365129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.365088 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.365129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.365120 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.365129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.365129 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a4be9-9f2a-403d-bd73-07845ae9eded-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.365129 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.365138 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjptm\" (UniqueName: \"kubernetes.io/projected/7d4a4be9-9f2a-403d-bd73-07845ae9eded-kube-api-access-sjptm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:33.837530 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:33.837489 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" podUID="b2a609c7-b0df-432a-90ff-541709a8b8ce" containerName="main" containerID="cri-o://05f7177185d556f8a8c1f6dcc454252bafea04ffc7aa463706e56b389c697fa0" gracePeriod=45 Jul 16 11:40:34.049633 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.049606 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" Jul 16 11:40:34.050132 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.049642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w" event={"ID":"7d4a4be9-9f2a-403d-bd73-07845ae9eded","Type":"ContainerDied","Data":"753b26b9fb3d54289d4faa522b0f7917c967378a56f958777a10a17dd949bfe3"} Jul 16 11:40:34.050132 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.049697 2580 scope.go:117] "RemoveContainer" containerID="0cd917aae653d33bcf404caa8435d3644e206513ce18459792be68544b05035f" Jul 16 11:40:34.051459 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.051431 2580 generic.go:356] "Generic (PLEG): container finished" podID="b2a609c7-b0df-432a-90ff-541709a8b8ce" containerID="05f7177185d556f8a8c1f6dcc454252bafea04ffc7aa463706e56b389c697fa0" exitCode=0 Jul 16 11:40:34.051565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.051502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" event={"ID":"b2a609c7-b0df-432a-90ff-541709a8b8ce","Type":"ContainerDied","Data":"05f7177185d556f8a8c1f6dcc454252bafea04ffc7aa463706e56b389c697fa0"} Jul 16 11:40:34.070948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.070918 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:40:34.074730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.074706 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a0660d/test-model-a0660d-kserve-57d748d99f-kqx6w"] Jul 16 11:40:34.094313 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.094256 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:34.171792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.171962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171826 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x489n\" (UniqueName: \"kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.171962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171846 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.171962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.171962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171920 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.171962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.171948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm\") pod \"b2a609c7-b0df-432a-90ff-541709a8b8ce\" (UID: \"b2a609c7-b0df-432a-90ff-541709a8b8ce\") " Jul 16 11:40:34.172241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.172115 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:34.172241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.172148 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home" (OuterVolumeSpecName: "home") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:34.172241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.172168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:34.172241 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.172227 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.172436 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.172249 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.174109 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.174079 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm" (OuterVolumeSpecName: "dshm") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:40:34.174109 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.174094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n" (OuterVolumeSpecName: "kube-api-access-x489n") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "kube-api-access-x489n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:40:34.174263 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.174142 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2a609c7-b0df-432a-90ff-541709a8b8ce" (UID: "b2a609c7-b0df-432a-90ff-541709a8b8ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:40:34.273654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.273600 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.273654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.273646 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a609c7-b0df-432a-90ff-541709a8b8ce-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.273654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.273655 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a609c7-b0df-432a-90ff-541709a8b8ce-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.273654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.273666 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x489n\" (UniqueName: \"kubernetes.io/projected/b2a609c7-b0df-432a-90ff-541709a8b8ce-kube-api-access-x489n\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:40:34.801960 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:34.801927 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" path="/var/lib/kubelet/pods/7d4a4be9-9f2a-403d-bd73-07845ae9eded/volumes" Jul 16 11:40:35.058744 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:35.058666 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" Jul 16 11:40:35.058744 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:35.058686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf" event={"ID":"b2a609c7-b0df-432a-90ff-541709a8b8ce","Type":"ContainerDied","Data":"78137a0262195eae89175180f55cf397c55c96c6641e08d45cc6a8884d7af594"} Jul 16 11:40:35.058744 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:35.058733 2580 scope.go:117] "RemoveContainer" containerID="05f7177185d556f8a8c1f6dcc454252bafea04ffc7aa463706e56b389c697fa0" Jul 16 11:40:35.076008 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:35.075978 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:40:35.079749 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:35.079719 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ca07b2/test-model-ca07b2-kserve-5f9586f9ff-7blwf"] Jul 16 11:40:36.802839 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:40:36.802807 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a609c7-b0df-432a-90ff-541709a8b8ce" path="/var/lib/kubelet/pods/b2a609c7-b0df-432a-90ff-541709a8b8ce/volumes" Jul 16 11:41:04.635096 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:41:04.635057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:41:16.642730 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:41:16.642696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:41:27.140246 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:41:27.140199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:41:39.530590 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:41:39.530547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:41:49.940844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:41:49.940801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:42:01.735180 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:01.735143 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:42:10.442356 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:10.442318 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:42:24.538285 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:24.538245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:42:47.030263 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030229 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030578 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" containerName="main" Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030590 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" containerName="main" Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030602 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a609c7-b0df-432a-90ff-541709a8b8ce" containerName="main" Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030608 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a609c7-b0df-432a-90ff-541709a8b8ce" containerName="main" Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030662 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4a4be9-9f2a-403d-bd73-07845ae9eded" containerName="main" Jul 16 11:42:47.030671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.030673 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a609c7-b0df-432a-90ff-541709a8b8ce" containerName="main" Jul 16 11:42:47.033569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.033546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.036078 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.036053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"kube-root-ca.crt\"" Jul 16 11:42:47.036874 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.036837 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"openshift-service-ca.crt\"" Jul 16 11:42:47.037051 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.037036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"default-dockercfg-4266x\"" Jul 16 11:42:47.037195 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.037176 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"auth-test-model-bfa604-kserve-self-signed-certs\"" Jul 16 11:42:47.044427 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.044403 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:42:47.136967 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.136930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.137171 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.137028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.137171 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.137068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.137171 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.137118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4brg\" (UniqueName: \"kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.137342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.137173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.137342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.137230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238479 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238479 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238743 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4brg\" (UniqueName: \"kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238743 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238743 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238743 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238909 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238948 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.238985 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.238964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.240811 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.240783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.241077 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.241058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.247185 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.247155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4brg\" (UniqueName: \"kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg\") pod \"auth-test-model-bfa604-kserve-b649b5447-8pwpv\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.345676 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.345566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:47.479092 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.479063 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:42:47.481293 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:42:47.481264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9af4883_2624_407b_bea2_0702af1fd5fe.slice/crio-cbd5b082f1b3dd4bdde3f031fe0e4bc716fab7234e63df4fb5a1610ee02fbff0 WatchSource:0}: Error finding container cbd5b082f1b3dd4bdde3f031fe0e4bc716fab7234e63df4fb5a1610ee02fbff0: Status 404 returned error can't find the container with id cbd5b082f1b3dd4bdde3f031fe0e4bc716fab7234e63df4fb5a1610ee02fbff0 Jul 16 11:42:47.698099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.698062 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" event={"ID":"b9af4883-2624-407b-bea2-0702af1fd5fe","Type":"ContainerStarted","Data":"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6"} Jul 16 11:42:47.698099 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.698105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" event={"ID":"b9af4883-2624-407b-bea2-0702af1fd5fe","Type":"ContainerStarted","Data":"cbd5b082f1b3dd4bdde3f031fe0e4bc716fab7234e63df4fb5a1610ee02fbff0"} Jul 16 11:42:47.718291 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:47.718226 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" podStartSLOduration=0.718209786 podStartE2EDuration="718.209786ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:42:47.715842276 +0000 UTC m=+2051.481975413" watchObservedRunningTime="2026-07-16 11:42:47.718209786 +0000 UTC m=+2051.484342922" Jul 16 11:42:51.059454 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:51.059414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:42:57.346397 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:57.346341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:57.346853 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:57.346422 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:57.358998 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:57.358972 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:42:57.758110 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:42:57.758070 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:43:03.826133 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.826094 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:43:03.829645 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.829627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.832090 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.832066 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"openshift-service-ca.crt\"" Jul 16 11:43:03.832887 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.832867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"kube-root-ca.crt\"" Jul 16 11:43:03.832984 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.832937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"default-dockercfg-w6ggr\"" Jul 16 11:43:03.833165 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.833146 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"auth-test-model-f05981-kserve-self-signed-certs\"" Jul 16 11:43:03.841497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.841472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:43:03.886244 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrkv\" (UniqueName: \"kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.886460 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.886460 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.886460 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.886460 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.886653 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.886507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987339 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987339 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrkv\" (UniqueName: \"kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987773 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987773 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.987980 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.988072 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.987981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.989724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.989706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.989994 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.989977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:03.996445 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:03.996413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrkv\" (UniqueName: \"kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv\") pod \"auth-test-model-f05981-kserve-7c57f77b95-bgf7k\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:04.139763 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:04.139718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:04.274498 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:04.274474 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:43:04.276387 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:43:04.276348 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfef714b_7cc2_41e8_aa0b_d4753fa88946.slice/crio-4290581f4e0ea2fa2aa622193d5534a7c32bd128958ad5c873d5c188620f5925 WatchSource:0}: Error finding container 4290581f4e0ea2fa2aa622193d5534a7c32bd128958ad5c873d5c188620f5925: Status 404 returned error can't find the container with id 4290581f4e0ea2fa2aa622193d5534a7c32bd128958ad5c873d5c188620f5925 Jul 16 11:43:04.782023 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:04.781975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" event={"ID":"cfef714b-7cc2-41e8-aa0b-d4753fa88946","Type":"ContainerStarted","Data":"c7effc28fa59098e54f51ddd5cc6901a2fb4029e69de72b54341f9ad07337a35"} Jul 16 11:43:04.782204 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:04.782034 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" event={"ID":"cfef714b-7cc2-41e8-aa0b-d4753fa88946","Type":"ContainerStarted","Data":"4290581f4e0ea2fa2aa622193d5534a7c32bd128958ad5c873d5c188620f5925"} Jul 16 11:43:04.798868 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:04.798801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" podStartSLOduration=1.7987850399999998 podStartE2EDuration="1.79878504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:43:04.798613541 +0000 UTC m=+2068.564746669" watchObservedRunningTime="2026-07-16 11:43:04.79878504 +0000 UTC m=+2068.564918176" Jul 16 11:43:07.630083 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:07.630047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:43:14.140870 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:14.140830 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:14.141422 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:14.140892 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:14.153539 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:14.153513 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:14.843963 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:14.843934 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:43:33.834772 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:33.834739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:43:36.830983 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:36.830952 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:43:36.837027 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:36.837001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:43:52.132599 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:43:52.132561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:44:06.238438 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:44:06.238401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:44:21.335780 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:44:21.335742 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:44:35.329148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:44:35.329062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:44:52.334242 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:44:52.334209 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:45:11.328844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:11.328806 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:45:14.400123 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:14.400084 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:45:15.309569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:15.309533 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:45:21.213320 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.213284 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:45:21.216611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.216591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.218930 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.218900 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"sub-test-model-bfa604-kserve-self-signed-certs\"" Jul 16 11:45:21.219598 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.219717 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.219717 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.219717 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.219900 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bwn\" (UniqueName: \"kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.219900 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.219763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.230175 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.230151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:45:21.321083 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321083 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321323 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321323 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321323 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bwn\" (UniqueName: \"kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321323 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321601 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321661 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.321661 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.321631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.323355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.323338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.323756 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.323737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.329117 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.329091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bwn\" (UniqueName: \"kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn\") pod \"sub-test-model-bfa604-kserve-8578875bf-m8chj\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.528402 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.528300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:21.661581 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:21.661551 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:45:21.663766 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:45:21.663734 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod012eb71e_b4ef_48c0_b6ed_ccc4925cf8ce.slice/crio-2b7bddaec0c0cf04ade143f9690b1c3d47493a9468e16514ae55944647851f84 WatchSource:0}: Error finding container 2b7bddaec0c0cf04ade143f9690b1c3d47493a9468e16514ae55944647851f84: Status 404 returned error can't find the container with id 2b7bddaec0c0cf04ade143f9690b1c3d47493a9468e16514ae55944647851f84 Jul 16 11:45:22.469856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:22.469819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" event={"ID":"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce","Type":"ContainerStarted","Data":"f8a05b7063cd4e494a45a2b922ccbf4321208be0d92dc9aa7b8d9579bf433332"} Jul 16 11:45:22.469856 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:22.469860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" event={"ID":"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce","Type":"ContainerStarted","Data":"2b7bddaec0c0cf04ade143f9690b1c3d47493a9468e16514ae55944647851f84"} Jul 16 11:45:28.929827 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:28.929771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" podStartSLOduration=7.929754642 podStartE2EDuration="7.929754642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:45:22.486407013 +0000 UTC m=+2206.252540152" watchObservedRunningTime="2026-07-16 11:45:28.929754642 +0000 UTC m=+2212.695887779" Jul 16 11:45:28.930334 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:28.930319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:45:29.421474 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.421438 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" podUID="b9af4883-2624-407b-bea2-0702af1fd5fe" containerName="main" containerID="cri-o://2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6" gracePeriod=45 Jul 16 11:45:29.671859 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.671797 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:45:29.695665 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695634 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.695851 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.695851 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695708 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.695851 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695765 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4brg\" (UniqueName: \"kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.695851 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695792 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.695851 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.695817 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home\") pod \"b9af4883-2624-407b-bea2-0702af1fd5fe\" (UID: \"b9af4883-2624-407b-bea2-0702af1fd5fe\") " Jul 16 11:45:29.696691 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.696406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home" (OuterVolumeSpecName: "home") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:29.696691 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.696518 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:29.696691 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.696651 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:29.698461 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.698435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:45:29.698570 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.698443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm" (OuterVolumeSpecName: "dshm") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:29.698570 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.698533 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg" (OuterVolumeSpecName: "kube-api-access-j4brg") pod "b9af4883-2624-407b-bea2-0702af1fd5fe" (UID: "b9af4883-2624-407b-bea2-0702af1fd5fe"). InnerVolumeSpecName "kube-api-access-j4brg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:45:29.797051 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797020 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:29.797051 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797047 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:29.797051 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797057 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9af4883-2624-407b-bea2-0702af1fd5fe-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:29.797274 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797067 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4brg\" (UniqueName: \"kubernetes.io/projected/b9af4883-2624-407b-bea2-0702af1fd5fe-kube-api-access-j4brg\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:29.797274 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797075 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:29.797274 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:29.797083 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9af4883-2624-407b-bea2-0702af1fd5fe-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.332122 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.332082 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" podUID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" containerName="main" containerID="cri-o://c7effc28fa59098e54f51ddd5cc6901a2fb4029e69de72b54341f9ad07337a35" gracePeriod=45 Jul 16 11:45:30.511611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.511566 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9af4883-2624-407b-bea2-0702af1fd5fe" containerID="2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6" exitCode=0 Jul 16 11:45:30.511611 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.511605 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" Jul 16 11:45:30.511857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.511645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" event={"ID":"b9af4883-2624-407b-bea2-0702af1fd5fe","Type":"ContainerDied","Data":"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6"} Jul 16 11:45:30.511857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.511692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv" event={"ID":"b9af4883-2624-407b-bea2-0702af1fd5fe","Type":"ContainerDied","Data":"cbd5b082f1b3dd4bdde3f031fe0e4bc716fab7234e63df4fb5a1610ee02fbff0"} Jul 16 11:45:30.511857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.511715 2580 scope.go:117] "RemoveContainer" containerID="2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6" Jul 16 11:45:30.513725 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.513703 2580 generic.go:356] "Generic (PLEG): container finished" podID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" containerID="c7effc28fa59098e54f51ddd5cc6901a2fb4029e69de72b54341f9ad07337a35" exitCode=0 Jul 16 11:45:30.513844 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.513777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" event={"ID":"cfef714b-7cc2-41e8-aa0b-d4753fa88946","Type":"ContainerDied","Data":"c7effc28fa59098e54f51ddd5cc6901a2fb4029e69de72b54341f9ad07337a35"} Jul 16 11:45:30.524819 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.524791 2580 scope.go:117] "RemoveContainer" containerID="2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6" Jul 16 11:45:30.525141 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:45:30.525119 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6\": container with ID starting with 2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6 not found: ID does not exist" containerID="2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6" Jul 16 11:45:30.525248 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.525149 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6"} err="failed to get container status \"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6\": rpc error: code = NotFound desc = could not find container \"2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6\": container with ID starting with 2b9d42ee89aad64fe5c96a3bf0f7e460bfd0fa76af25b42ccfcded96a68d6fd6 not found: ID does not exist" Jul 16 11:45:30.534806 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.534777 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:45:30.538746 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.538723 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/auth-test-model-bfa604-kserve-b649b5447-8pwpv"] Jul 16 11:45:30.587431 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.587342 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:45:30.705528 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705557 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705622 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705639 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsrkv\" (UniqueName: \"kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705656 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm\") pod \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\" (UID: \"cfef714b-7cc2-41e8-aa0b-d4753fa88946\") " Jul 16 11:45:30.705986 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:30.705986 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705858 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:30.705986 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.705909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home" (OuterVolumeSpecName: "home") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:30.707932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.707901 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm" (OuterVolumeSpecName: "dshm") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:45:30.708042 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.707960 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:45:30.708042 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.708004 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv" (OuterVolumeSpecName: "kube-api-access-jsrkv") pod "cfef714b-7cc2-41e8-aa0b-d4753fa88946" (UID: "cfef714b-7cc2-41e8-aa0b-d4753fa88946"). InnerVolumeSpecName "kube-api-access-jsrkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:45:30.802742 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.802711 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9af4883-2624-407b-bea2-0702af1fd5fe" path="/var/lib/kubelet/pods/b9af4883-2624-407b-bea2-0702af1fd5fe/volumes" Jul 16 11:45:30.806708 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806690 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.806774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806711 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrkv\" (UniqueName: \"kubernetes.io/projected/cfef714b-7cc2-41e8-aa0b-d4753fa88946-kube-api-access-jsrkv\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.806774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806721 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.806774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806729 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.806774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806738 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfef714b-7cc2-41e8-aa0b-d4753fa88946-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:30.806774 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:30.806746 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfef714b-7cc2-41e8-aa0b-d4753fa88946-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:45:31.520995 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.520958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" event={"ID":"cfef714b-7cc2-41e8-aa0b-d4753fa88946","Type":"ContainerDied","Data":"4290581f4e0ea2fa2aa622193d5534a7c32bd128958ad5c873d5c188620f5925"} Jul 16 11:45:31.521439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.520999 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k" Jul 16 11:45:31.521439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.521011 2580 scope.go:117] "RemoveContainer" containerID="c7effc28fa59098e54f51ddd5cc6901a2fb4029e69de72b54341f9ad07337a35" Jul 16 11:45:31.528724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.528696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:31.528852 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.528741 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:31.538441 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.538410 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:45:31.541781 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.541760 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:31.544568 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:31.544542 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/auth-test-model-f05981-kserve-7c57f77b95-bgf7k"] Jul 16 11:45:32.538137 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:32.538109 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:45:32.802355 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:32.802267 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" path="/var/lib/kubelet/pods/cfef714b-7cc2-41e8-aa0b-d4753fa88946/volumes" Jul 16 11:45:56.127503 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127464 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127789 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" containerName="main" Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127799 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" containerName="main" Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127811 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af4883-2624-407b-bea2-0702af1fd5fe" containerName="main" Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127816 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af4883-2624-407b-bea2-0702af1fd5fe" containerName="main" Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127880 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfef714b-7cc2-41e8-aa0b-d4753fa88946" containerName="main" Jul 16 11:45:56.127990 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.127892 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af4883-2624-407b-bea2-0702af1fd5fe" containerName="main" Jul 16 11:45:56.130889 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.130872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.133455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.133423 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"sub-test-model-f05981-kserve-self-signed-certs\"" Jul 16 11:45:56.133455 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.133442 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"openshift-service-ca.crt\"" Jul 16 11:45:56.134224 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.134207 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"kube-root-ca.crt\"" Jul 16 11:45:56.134527 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.134512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"default-dockercfg-w6ggr\"" Jul 16 11:45:56.140720 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.140692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:45:56.235245 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.235245 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdzd\" (UniqueName: \"kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.235532 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.235532 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.235532 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.235532 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.235487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.336977 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.336936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdzd\" (UniqueName: \"kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337172 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337452 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337575 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.337620 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.337559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.339329 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.339311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.339642 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.339624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.350767 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.350739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdzd\" (UniqueName: \"kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd\") pod \"sub-test-model-f05981-kserve-689f5b556f-vtd6h\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.442155 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.442062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:45:56.588293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.588264 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:45:56.590763 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:45:56.590733 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95bc8351_84f4_44f6_8114_d141150fc26e.slice/crio-6dd82c251ca6ab1aa387b2d63b725c83d6a8a97521044a42547d190575b2b2e8 WatchSource:0}: Error finding container 6dd82c251ca6ab1aa387b2d63b725c83d6a8a97521044a42547d190575b2b2e8: Status 404 returned error can't find the container with id 6dd82c251ca6ab1aa387b2d63b725c83d6a8a97521044a42547d190575b2b2e8 Jul 16 11:45:56.641511 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:56.641488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" event={"ID":"95bc8351-84f4-44f6-8114-d141150fc26e","Type":"ContainerStarted","Data":"6dd82c251ca6ab1aa387b2d63b725c83d6a8a97521044a42547d190575b2b2e8"} Jul 16 11:45:57.647943 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:57.647862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" event={"ID":"95bc8351-84f4-44f6-8114-d141150fc26e","Type":"ContainerStarted","Data":"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7"} Jul 16 11:45:57.667879 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:45:57.667835 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" podStartSLOduration=1.667819296 podStartE2EDuration="1.667819296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:45:57.66651222 +0000 UTC m=+2241.432645368" watchObservedRunningTime="2026-07-16 11:45:57.667819296 +0000 UTC m=+2241.433952432" Jul 16 11:46:00.235614 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:00.235573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:46:06.442759 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:06.442724 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:06.443202 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:06.443003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:06.455889 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:06.455862 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:06.705743 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:06.705644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:19.858596 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:19.858559 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:46:35.729143 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:35.729106 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:46:36.624954 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:36.624918 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:46:40.540234 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:40.540125 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:46:50.749086 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:50.749032 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" podUID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" containerName="main" containerID="cri-o://f8a05b7063cd4e494a45a2b922ccbf4321208be0d92dc9aa7b8d9579bf433332" gracePeriod=45 Jul 16 11:46:50.912683 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:50.912640 2580 generic.go:356] "Generic (PLEG): container finished" podID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" containerID="f8a05b7063cd4e494a45a2b922ccbf4321208be0d92dc9aa7b8d9579bf433332" exitCode=0 Jul 16 11:46:50.912852 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:50.912700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" event={"ID":"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce","Type":"ContainerDied","Data":"f8a05b7063cd4e494a45a2b922ccbf4321208be0d92dc9aa7b8d9579bf433332"} Jul 16 11:46:51.002203 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.002132 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:46:51.034741 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.034708 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.034741 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.034748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9bwn\" (UniqueName: \"kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.034987 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.034776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.034987 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.034956 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.035097 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.035097 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035048 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.035205 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm\") pod \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\" (UID: \"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce\") " Jul 16 11:46:51.035258 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.035258 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home" (OuterVolumeSpecName: "home") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.035518 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035493 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.035518 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035518 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.035715 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.035533 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.037058 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.037034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:46:51.037153 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.037093 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn" (OuterVolumeSpecName: "kube-api-access-k9bwn") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "kube-api-access-k9bwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:46:51.037424 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.037350 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm" (OuterVolumeSpecName: "dshm") pod "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" (UID: "012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.136226 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.136178 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.136226 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.136214 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9bwn\" (UniqueName: \"kubernetes.io/projected/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-kube-api-access-k9bwn\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.136226 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.136225 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.644778 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.644714 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" podUID="95bc8351-84f4-44f6-8114-d141150fc26e" containerName="main" containerID="cri-o://21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7" gracePeriod=45 Jul 16 11:46:51.893207 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.893178 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:51.919148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.919069 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" Jul 16 11:46:51.919148 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.919118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj" event={"ID":"012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce","Type":"ContainerDied","Data":"2b7bddaec0c0cf04ade143f9690b1c3d47493a9468e16514ae55944647851f84"} Jul 16 11:46:51.919456 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.919161 2580 scope.go:117] "RemoveContainer" containerID="f8a05b7063cd4e494a45a2b922ccbf4321208be0d92dc9aa7b8d9579bf433332" Jul 16 11:46:51.925167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.925134 2580 generic.go:356] "Generic (PLEG): container finished" podID="95bc8351-84f4-44f6-8114-d141150fc26e" containerID="21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7" exitCode=0 Jul 16 11:46:51.925314 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.925191 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" Jul 16 11:46:51.925314 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.925192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" event={"ID":"95bc8351-84f4-44f6-8114-d141150fc26e","Type":"ContainerDied","Data":"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7"} Jul 16 11:46:51.925314 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.925307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h" event={"ID":"95bc8351-84f4-44f6-8114-d141150fc26e","Type":"ContainerDied","Data":"6dd82c251ca6ab1aa387b2d63b725c83d6a8a97521044a42547d190575b2b2e8"} Jul 16 11:46:51.933912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.933896 2580 scope.go:117] "RemoveContainer" containerID="21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7" Jul 16 11:46:51.941919 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.941892 2580 scope.go:117] "RemoveContainer" containerID="21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7" Jul 16 11:46:51.942237 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:46:51.942211 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7\": container with ID starting with 21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7 not found: ID does not exist" containerID="21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7" Jul 16 11:46:51.942297 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.942251 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7"} err="failed to get container status \"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7\": rpc error: code = NotFound desc = could not find container \"21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7\": container with ID starting with 21af73556ec4ad14d4381600265caa6b0f669cd47a0c13ce1d1b84229db47fa7 not found: ID does not exist" Jul 16 11:46:51.943474 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943453 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943564 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943619 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943584 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmdzd\" (UniqueName: \"kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943619 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943603 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943714 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943619 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943714 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm\") pod \"95bc8351-84f4-44f6-8114-d141150fc26e\" (UID: \"95bc8351-84f4-44f6-8114-d141150fc26e\") " Jul 16 11:46:51.943798 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.943954 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943931 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home" (OuterVolumeSpecName: "home") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.944003 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943950 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:51.944003 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.943959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache" (OuterVolumeSpecName: "model-cache") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.945875 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.945847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm" (OuterVolumeSpecName: "dshm") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:46:51.946105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.945999 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd" (OuterVolumeSpecName: "kube-api-access-nmdzd") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "kube-api-access-nmdzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:46:51.946105 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.946075 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95bc8351-84f4-44f6-8114-d141150fc26e" (UID: "95bc8351-84f4-44f6-8114-d141150fc26e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:46:51.948021 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.948000 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:46:51.953830 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:51.953808 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/sub-test-model-bfa604-kserve-8578875bf-m8chj"] Jul 16 11:46:52.045040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.045010 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bc8351-84f4-44f6-8114-d141150fc26e-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:52.045040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.045036 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdzd\" (UniqueName: \"kubernetes.io/projected/95bc8351-84f4-44f6-8114-d141150fc26e-kube-api-access-nmdzd\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:52.045040 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.045047 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:52.045273 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.045056 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:52.045273 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.045067 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bc8351-84f4-44f6-8114-d141150fc26e-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:46:52.247079 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.247047 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:46:52.251860 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.251835 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/sub-test-model-f05981-kserve-689f5b556f-vtd6h"] Jul 16 11:46:52.807293 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.807250 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" path="/var/lib/kubelet/pods/012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce/volumes" Jul 16 11:46:52.807748 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:46:52.807728 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95bc8351-84f4-44f6-8114-d141150fc26e" path="/var/lib/kubelet/pods/95bc8351-84f4-44f6-8114-d141150fc26e/volumes" Jul 16 11:47:03.597622 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.597592 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.597939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bc8351-84f4-44f6-8114-d141150fc26e" containerName="main" Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.597953 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bc8351-84f4-44f6-8114-d141150fc26e" containerName="main" Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.597970 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" containerName="main" Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.597975 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" containerName="main" Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.598032 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="012eb71e-b4ef-48c0-b6ed-ccc4925cf8ce" containerName="main" Jul 16 11:47:03.598167 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.598043 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bc8351-84f4-44f6-8114-d141150fc26e" containerName="main" Jul 16 11:47:03.602655 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.602636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.605671 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.605649 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"rate-test-model-bfa604-kserve-self-signed-certs\"" Jul 16 11:47:03.605893 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.605872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"default-dockercfg-4266x\"" Jul 16 11:47:03.605949 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.605878 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"openshift-service-ca.crt\"" Jul 16 11:47:03.605996 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.605950 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bfa604\"/\"kube-root-ca.crt\"" Jul 16 11:47:03.610846 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.610824 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:47:03.643497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmxn\" (UniqueName: \"kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.643667 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.643667 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.643667 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.643792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.643792 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.643738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744594 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744594 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744840 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmxn\" (UniqueName: \"kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744840 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744840 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.744840 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.744729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.745091 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.745065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.745128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.745092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.745260 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.745246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.746971 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.746948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.747234 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.747216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.754686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.754656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmxn\" (UniqueName: \"kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn\") pod \"rate-test-model-bfa604-kserve-955f45f95-m26l9\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:03.912578 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:03.912544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:04.049524 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:04.049498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:47:04.051392 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:47:04.051347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d706761_99b3_4445_95c4_74826f90e388.slice/crio-d1aefc32d21f6fadfe58b2808661fe0a69562948473a3fe5684f8b0cec0f603c WatchSource:0}: Error finding container d1aefc32d21f6fadfe58b2808661fe0a69562948473a3fe5684f8b0cec0f603c: Status 404 returned error can't find the container with id d1aefc32d21f6fadfe58b2808661fe0a69562948473a3fe5684f8b0cec0f603c Jul 16 11:47:04.992641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:04.992607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" event={"ID":"4d706761-99b3-4445-95c4-74826f90e388","Type":"ContainerStarted","Data":"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d"} Jul 16 11:47:04.992641 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:04.992647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" event={"ID":"4d706761-99b3-4445-95c4-74826f90e388","Type":"ContainerStarted","Data":"d1aefc32d21f6fadfe58b2808661fe0a69562948473a3fe5684f8b0cec0f603c"} Jul 16 11:47:05.012771 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:05.012721 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" podStartSLOduration=2.01270504 podStartE2EDuration="2.01270504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:47:05.010784786 +0000 UTC m=+2308.776917923" watchObservedRunningTime="2026-07-16 11:47:05.01270504 +0000 UTC m=+2308.778838218" Jul 16 11:47:07.229443 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:07.229403 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:47:13.912750 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:13.912707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:13.913202 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:13.913026 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:13.925263 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:13.925237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:14.048932 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:14.048902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:47:33.921928 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.921844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:47:33.925435 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.925415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:33.928570 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.928533 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"kube-root-ca.crt\"" Jul 16 11:47:33.928722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.928538 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"openshift-service-ca.crt\"" Jul 16 11:47:33.928722 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.928687 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"rate-test-model-f05981-kserve-self-signed-certs\"" Jul 16 11:47:33.928858 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.928696 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f05981\"/\"default-dockercfg-w6ggr\"" Jul 16 11:47:33.938439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:33.938414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:47:34.014009 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.013964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.014188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.014029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.014188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.014054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.014188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.014105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.014188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.014140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.014188 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.014159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4gm\" (UniqueName: \"kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.114837 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.114799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.114852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.114870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4gm\" (UniqueName: \"kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.114923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.114978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.115004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115348 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.115283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115348 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.115328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.115526 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.115357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.117255 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.117236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.117530 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.117512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.121959 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.121933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4gm\" (UniqueName: \"kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm\") pod \"rate-test-model-f05981-kserve-67687d7bc5-v9dtk\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.235543 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.235438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:34.375122 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:34.375087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:47:34.377147 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:47:34.377113 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d2e0f74_9ac7_4679_8500_9a8d0b0458f4.slice/crio-f2e5cee045414f74ba859e35c5ad6f7cce5225d2e409341e2be68bc205b1ad2b WatchSource:0}: Error finding container f2e5cee045414f74ba859e35c5ad6f7cce5225d2e409341e2be68bc205b1ad2b: Status 404 returned error can't find the container with id f2e5cee045414f74ba859e35c5ad6f7cce5225d2e409341e2be68bc205b1ad2b Jul 16 11:47:35.143553 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:35.143508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" event={"ID":"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4","Type":"ContainerStarted","Data":"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff"} Jul 16 11:47:35.143957 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:35.143561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" event={"ID":"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4","Type":"ContainerStarted","Data":"f2e5cee045414f74ba859e35c5ad6f7cce5225d2e409341e2be68bc205b1ad2b"} Jul 16 11:47:35.162411 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:35.162339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" podStartSLOduration=2.162323427 podStartE2EDuration="2.162323427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 11:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:47:35.160856644 +0000 UTC m=+2338.926989782" watchObservedRunningTime="2026-07-16 11:47:35.162323427 +0000 UTC m=+2338.928456543" Jul 16 11:47:38.337283 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:38.337243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:47:44.236447 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:44.236408 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:44.236939 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:44.236494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:44.248902 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:44.248874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:45.209753 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:45.209723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:47:58.933155 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:47:58.933120 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:48:13.447760 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:13.447726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:48:20.437409 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:20.437347 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:48:21.353514 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:21.353486 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:48:27.528777 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:27.528739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:48:35.457088 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.457049 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" podUID="4d706761-99b3-4445-95c4-74826f90e388" containerName="main" containerID="cri-o://052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d" gracePeriod=45 Jul 16 11:48:35.703564 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.703538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:48:35.754580 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754580 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754550 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lmxn\" (UniqueName: \"kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754580 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754573 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754612 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754640 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754683 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm\") pod \"4d706761-99b3-4445-95c4-74826f90e388\" (UID: \"4d706761-99b3-4445-95c4-74826f90e388\") " Jul 16 11:48:35.754871 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754846 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:35.755090 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.754858 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home" (OuterVolumeSpecName: "home") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:35.755090 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.755026 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:35.756848 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.756823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn" (OuterVolumeSpecName: "kube-api-access-2lmxn") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "kube-api-access-2lmxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:48:35.756848 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.756824 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm" (OuterVolumeSpecName: "dshm") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:35.756993 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.756848 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d706761-99b3-4445-95c4-74826f90e388" (UID: "4d706761-99b3-4445-95c4-74826f90e388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:48:35.856316 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856280 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:35.856316 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856308 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:35.856316 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856321 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmxn\" (UniqueName: \"kubernetes.io/projected/4d706761-99b3-4445-95c4-74826f90e388-kube-api-access-2lmxn\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:35.856576 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856330 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:35.856576 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856339 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d706761-99b3-4445-95c4-74826f90e388-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:35.856576 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:35.856346 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d706761-99b3-4445-95c4-74826f90e388-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.374401 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.374342 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" podUID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" containerName="main" containerID="cri-o://3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff" gracePeriod=45 Jul 16 11:48:36.446202 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.446168 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d706761-99b3-4445-95c4-74826f90e388" containerID="052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d" exitCode=0 Jul 16 11:48:36.446426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.446220 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" Jul 16 11:48:36.446426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.446235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" event={"ID":"4d706761-99b3-4445-95c4-74826f90e388","Type":"ContainerDied","Data":"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d"} Jul 16 11:48:36.446426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.446265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9" event={"ID":"4d706761-99b3-4445-95c4-74826f90e388","Type":"ContainerDied","Data":"d1aefc32d21f6fadfe58b2808661fe0a69562948473a3fe5684f8b0cec0f603c"} Jul 16 11:48:36.446426 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.446280 2580 scope.go:117] "RemoveContainer" containerID="052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d" Jul 16 11:48:36.467962 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.467931 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:48:36.471690 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.471666 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bfa604/rate-test-model-bfa604-kserve-955f45f95-m26l9"] Jul 16 11:48:36.486017 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.485932 2580 scope.go:117] "RemoveContainer" containerID="052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d" Jul 16 11:48:36.486345 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:48:36.486323 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d\": container with ID starting with 052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d not found: ID does not exist" containerID="052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d" Jul 16 11:48:36.486466 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.486355 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d"} err="failed to get container status \"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d\": rpc error: code = NotFound desc = could not find container \"052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d\": container with ID starting with 052b50065fb30f5ab69164fdada787704e905bacae21071bc5f6749e6467172d not found: ID does not exist" Jul 16 11:48:36.633748 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.633674 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:48:36.662729 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.662912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662740 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.662912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662813 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.662912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662842 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s4gm\" (UniqueName: \"kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.662912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662867 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.663107 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.662937 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm\") pod \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\" (UID: \"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4\") " Jul 16 11:48:36.663107 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663001 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home" (OuterVolumeSpecName: "home") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:36.663107 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663013 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:36.663107 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663086 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:36.663569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663210 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-model-cache\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.663569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663226 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-home\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.663569 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.663239 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tmp-dir\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.665228 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.665203 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 11:48:36.665342 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.665269 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm" (OuterVolumeSpecName: "kube-api-access-4s4gm") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "kube-api-access-4s4gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:48:36.665465 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.665348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm" (OuterVolumeSpecName: "dshm") pod "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" (UID: "6d2e0f74-9ac7-4679-8500-9a8d0b0458f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 11:48:36.764373 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.764336 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4gm\" (UniqueName: \"kubernetes.io/projected/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-kube-api-access-4s4gm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.764373 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.764367 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-tls-certs\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.764373 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.764400 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4-dshm\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:36.803440 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.803403 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d706761-99b3-4445-95c4-74826f90e388" path="/var/lib/kubelet/pods/4d706761-99b3-4445-95c4-74826f90e388/volumes" Jul 16 11:48:36.862765 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.862741 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:48:36.870359 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:36.870341 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vrmqc_1c15c583-3e8a-4ea6-9cf4-45c3702d0517/ovn-acl-logging/0.log" Jul 16 11:48:37.452530 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.452494 2580 generic.go:356] "Generic (PLEG): container finished" podID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" containerID="3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff" exitCode=0 Jul 16 11:48:37.452724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.452538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" Jul 16 11:48:37.452724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.452578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" event={"ID":"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4","Type":"ContainerDied","Data":"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff"} Jul 16 11:48:37.452724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.452618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk" event={"ID":"6d2e0f74-9ac7-4679-8500-9a8d0b0458f4","Type":"ContainerDied","Data":"f2e5cee045414f74ba859e35c5ad6f7cce5225d2e409341e2be68bc205b1ad2b"} Jul 16 11:48:37.452724 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.452638 2580 scope.go:117] "RemoveContainer" containerID="3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff" Jul 16 11:48:37.461224 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.461203 2580 scope.go:117] "RemoveContainer" containerID="3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff" Jul 16 11:48:37.461505 ip-10-0-130-16 kubenswrapper[2580]: E0716 11:48:37.461484 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff\": container with ID starting with 3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff not found: ID does not exist" containerID="3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff" Jul 16 11:48:37.461589 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.461511 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff"} err="failed to get container status \"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff\": rpc error: code = NotFound desc = could not find container \"3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff\": container with ID starting with 3e4d56908767fc0c51b6ac6f29618b5889c962a25a2ce31aa7ad118f965a34ff not found: ID does not exist" Jul 16 11:48:37.469615 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.469589 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:48:37.473230 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.473207 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f05981/rate-test-model-f05981-kserve-67687d7bc5-v9dtk"] Jul 16 11:48:37.668834 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.668799 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7375-139995116098160"] Jul 16 11:48:37.669142 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669130 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d706761-99b3-4445-95c4-74826f90e388" containerName="main" Jul 16 11:48:37.669186 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669143 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d706761-99b3-4445-95c4-74826f90e388" containerName="main" Jul 16 11:48:37.669186 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669154 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" containerName="main" Jul 16 11:48:37.669186 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669160 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" containerName="main" Jul 16 11:48:37.669278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669213 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" containerName="main" Jul 16 11:48:37.669278 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.669225 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d706761-99b3-4445-95c4-74826f90e388" containerName="main" Jul 16 11:48:37.673496 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.673466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:37.675808 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.675789 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x5wjq\"" Jul 16 11:48:37.679936 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.679915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-139995116098160"] Jul 16 11:48:37.771681 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.771586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsw4t\" (UniqueName: \"kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t\") pod \"test-curl-7375-139995116098160\" (UID: \"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec\") " pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:37.872439 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.872401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsw4t\" (UniqueName: \"kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t\") pod \"test-curl-7375-139995116098160\" (UID: \"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec\") " pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:37.880897 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.880862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsw4t\" (UniqueName: \"kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t\") pod \"test-curl-7375-139995116098160\" (UID: \"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec\") " pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:37.983212 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:37.983179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:38.114388 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:38.114347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-139995116098160"] Jul 16 11:48:38.116420 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:48:38.116389 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e7f00cf_2a3e_4d02_9d4d_6c8b8fcd00ec.slice/crio-fe55b02b70e7cc3c780b9add82228d2640fc691e010bf8f5969bda98eb424a0e WatchSource:0}: Error finding container fe55b02b70e7cc3c780b9add82228d2640fc691e010bf8f5969bda98eb424a0e: Status 404 returned error can't find the container with id fe55b02b70e7cc3c780b9add82228d2640fc691e010bf8f5969bda98eb424a0e Jul 16 11:48:38.117688 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:38.117667 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 11:48:38.461082 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:38.461052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-139995116098160" event={"ID":"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec","Type":"ContainerStarted","Data":"fe55b02b70e7cc3c780b9add82228d2640fc691e010bf8f5969bda98eb424a0e"} Jul 16 11:48:38.809686 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:38.809657 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2e0f74-9ac7-4679-8500-9a8d0b0458f4" path="/var/lib/kubelet/pods/6d2e0f74-9ac7-4679-8500-9a8d0b0458f4/volumes" Jul 16 11:48:39.470291 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:39.470255 2580 generic.go:356] "Generic (PLEG): container finished" podID="9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" containerID="1bd243daa87d2386386f9eb142734365a4453d0fd3f81ed801cb8673a87ce354" exitCode=0 Jul 16 11:48:39.470291 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:39.470284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-139995116098160" event={"ID":"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec","Type":"ContainerDied","Data":"1bd243daa87d2386386f9eb142734365a4453d0fd3f81ed801cb8673a87ce354"} Jul 16 11:48:40.601965 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.601932 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:40.640298 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.640272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7375-139995116098160_9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec/test-curl-7375-139995116098160/0.log" Jul 16 11:48:40.662150 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.662121 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7375-139995116098160"] Jul 16 11:48:40.666088 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.666064 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7375-139995116098160"] Jul 16 11:48:40.697877 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.697838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsw4t\" (UniqueName: \"kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t\") pod \"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec\" (UID: \"9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec\") " Jul 16 11:48:40.700094 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.700065 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t" (OuterVolumeSpecName: "kube-api-access-qsw4t") pod "9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" (UID: "9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec"). InnerVolumeSpecName "kube-api-access-qsw4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 11:48:40.799240 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.799149 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsw4t\" (UniqueName: \"kubernetes.io/projected/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec-kube-api-access-qsw4t\") on node \"ip-10-0-130-16.ec2.internal\" DevicePath \"\"" Jul 16 11:48:40.802527 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:40.802499 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" path="/var/lib/kubelet/pods/9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec/volumes" Jul 16 11:48:41.480567 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:41.480540 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-139995116098160" Jul 16 11:48:41.480567 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:41.480570 2580 scope.go:117] "RemoveContainer" containerID="1bd243daa87d2386386f9eb142734365a4453d0fd3f81ed801cb8673a87ce354" Jul 16 11:48:50.832486 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:48:50.832445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:49:03.833737 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:03.833626 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l7v5k"] Jul 16 11:49:07.133128 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:07.133093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cc598755-jksjs_88136d32-8887-433c-bf8e-9a768adb9386/authorino/0.log" Jul 16 11:49:13.146994 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:13.146968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8c4d5c89-drz7j_829b6646-37b1-48b7-9806-870dd3a62cd6/copy-manifests/0.log" Jul 16 11:49:13.158650 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:13.158618 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8c4d5c89-drz7j_829b6646-37b1-48b7-9806-870dd3a62cd6/manager/0.log" Jul 16 11:49:14.721993 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.721964 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/util/0.log" Jul 16 11:49:14.728084 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.728060 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/pull/0.log" Jul 16 11:49:14.733468 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.733437 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/extract/0.log" Jul 16 11:49:14.842857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.842828 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/util/0.log" Jul 16 11:49:14.848881 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.848856 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/pull/0.log" Jul 16 11:49:14.854857 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.854836 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/extract/0.log" Jul 16 11:49:14.958434 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.958368 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/util/0.log" Jul 16 11:49:14.964254 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.964232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/pull/0.log" Jul 16 11:49:14.969707 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:14.969686 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/extract/0.log" Jul 16 11:49:15.077617 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.077544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/util/0.log" Jul 16 11:49:15.085280 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.085250 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/pull/0.log" Jul 16 11:49:15.091133 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.091088 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/extract/0.log" Jul 16 11:49:15.210273 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.210243 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cc598755-jksjs_88136d32-8887-433c-bf8e-9a768adb9386/authorino/0.log" Jul 16 11:49:15.544275 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.544249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-m7bz9_1e8a7b3c-42f3-4971-946c-c42bf2eaff4e/kuadrant-console-plugin/0.log" Jul 16 11:49:15.905434 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:15.905406 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l7v5k_69bc7662-e7fd-489b-9853-8f2b91162b81/limitador/0.log" Jul 16 11:49:16.723665 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:16.723634 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79cdcbc7df-w8bzg_4e0f2523-eb5b-4fa4-a7df-769a9c01aae5/kube-auth-proxy/0.log" Jul 16 11:49:17.891314 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:17.891280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg65d_5e21a10d-4961-4dff-bb7e-150e21305a16/main/0.log" Jul 16 11:49:18.116643 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:18.116614 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-qbcz5_d9916b08-432a-455d-8acd-5b4140719ee5/main/0.log" Jul 16 11:49:26.553992 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:26.553958 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jkddl_ee647ad6-cd55-4717-b5ac-618f94505d75/global-pull-secret-syncer/0.log" Jul 16 11:49:26.716394 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:26.716351 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-szwbp_444e123d-6ada-4f32-8754-d805cb14097a/konnectivity-agent/0.log" Jul 16 11:49:26.788654 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:26.788623 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-16.ec2.internal_275e2b67084dfc9c3fa0f7611e1a99f6/haproxy/0.log" Jul 16 11:49:32.068282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.068249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/extract/0.log" Jul 16 11:49:32.105160 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.105130 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/util/0.log" Jul 16 11:49:32.132221 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.132192 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592qz8x_f2643461-6c50-43f3-a2ee-46b0af0b917d/pull/0.log" Jul 16 11:49:32.161514 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.161483 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/extract/0.log" Jul 16 11:49:32.184344 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.184313 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/util/0.log" Jul 16 11:49:32.207551 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.207522 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nh2sq_1e66f54a-3237-48f4-be15-07869e2325c4/pull/0.log" Jul 16 11:49:32.235372 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.235345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/extract/0.log" Jul 16 11:49:32.258245 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.258218 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/util/0.log" Jul 16 11:49:32.281176 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.281153 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c6s42_7b470038-adae-4829-a1d8-30ac699ab60b/pull/0.log" Jul 16 11:49:32.309549 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.309525 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/extract/0.log" Jul 16 11:49:32.334873 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.334788 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/util/0.log" Jul 16 11:49:32.363659 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.363630 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dblhs_4c7d24bd-bd96-4bd4-938f-4678b9687f2b/pull/0.log" Jul 16 11:49:32.652264 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.652226 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cc598755-jksjs_88136d32-8887-433c-bf8e-9a768adb9386/authorino/0.log" Jul 16 11:49:32.761594 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.761562 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-m7bz9_1e8a7b3c-42f3-4971-946c-c42bf2eaff4e/kuadrant-console-plugin/0.log" Jul 16 11:49:32.975307 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:32.975202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l7v5k_69bc7662-e7fd-489b-9853-8f2b91162b81/limitador/0.log" Jul 16 11:49:34.774982 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.774956 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/alertmanager/0.log" Jul 16 11:49:34.811737 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.811708 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/config-reloader/0.log" Jul 16 11:49:34.854847 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.854816 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/kube-rbac-proxy-web/0.log" Jul 16 11:49:34.891800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.891772 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/kube-rbac-proxy/0.log" Jul 16 11:49:34.927605 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.927574 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/kube-rbac-proxy-metric/0.log" Jul 16 11:49:34.958711 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.958679 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/prom-label-proxy/0.log" Jul 16 11:49:34.989869 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:34.989823 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e24b1558-ad87-4987-bf2d-787dfb72bb13/init-config-reloader/0.log" Jul 16 11:49:35.355670 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:35.355643 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb6t9_5c5ed361-19c0-4de3-8efa-0ea720588bd4/node-exporter/0.log" Jul 16 11:49:35.389822 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:35.389791 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb6t9_5c5ed361-19c0-4de3-8efa-0ea720588bd4/kube-rbac-proxy/0.log" Jul 16 11:49:35.425485 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:35.425459 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb6t9_5c5ed361-19c0-4de3-8efa-0ea720588bd4/init-textfile/0.log" Jul 16 11:49:35.992300 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:35.992219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-9tqkw_127e89ac-fe5a-4471-a90e-283f0d80aed9/prometheus-operator/0.log" Jul 16 11:49:36.032659 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:36.032619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-9tqkw_127e89ac-fe5a-4471-a90e-283f0d80aed9/kube-rbac-proxy/0.log" Jul 16 11:49:36.068805 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:36.068771 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-k2kmv_5ed9f36b-ee8c-4c10-a052-b5cea7a930c4/prometheus-operator-admission-webhook/0.log" Jul 16 11:49:37.079282 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.079252 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb"] Jul 16 11:49:37.079693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.079597 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" containerName="test-curl-7375-139995116098160" Jul 16 11:49:37.079693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.079609 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" containerName="test-curl-7375-139995116098160" Jul 16 11:49:37.079693 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.079667 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7f00cf-2a3e-4d02-9d4d-6c8b8fcd00ec" containerName="test-curl-7375-139995116098160" Jul 16 11:49:37.083912 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.083887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.086074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.086048 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p4bp\"/\"kube-root-ca.crt\"" Jul 16 11:49:37.086074 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.086066 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p4bp\"/\"openshift-service-ca.crt\"" Jul 16 11:49:37.086235 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.086066 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8p4bp\"/\"default-dockercfg-8p8z5\"" Jul 16 11:49:37.089367 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.089345 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb"] Jul 16 11:49:37.185675 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.185635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-proc\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.185675 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.185680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-podres\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.185906 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.185738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-sys\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.185906 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.185771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-lib-modules\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.185906 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.185805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvbf\" (UniqueName: \"kubernetes.io/projected/0acfeab3-7020-4791-b3e9-da6d0785fe5a-kube-api-access-bnvbf\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286666 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-sys\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286666 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-lib-modules\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286666 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvbf\" (UniqueName: \"kubernetes.io/projected/0acfeab3-7020-4791-b3e9-da6d0785fe5a-kube-api-access-bnvbf\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-proc\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-podres\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-sys\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-podres\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-proc\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.286929 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.286888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0acfeab3-7020-4791-b3e9-da6d0785fe5a-lib-modules\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.295048 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.295024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvbf\" (UniqueName: \"kubernetes.io/projected/0acfeab3-7020-4791-b3e9-da6d0785fe5a-kube-api-access-bnvbf\") pod \"perf-node-gather-daemonset-9klcb\" (UID: \"0acfeab3-7020-4791-b3e9-da6d0785fe5a\") " pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.395073 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.395038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.524927 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.524904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb"] Jul 16 11:49:37.526606 ip-10-0-130-16 kubenswrapper[2580]: W0716 11:49:37.526574 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0acfeab3_7020_4791_b3e9_da6d0785fe5a.slice/crio-ff14a7d361472b00ed67681698a55f587a5e5ae74bf5004aab1334aeac1f00a4 WatchSource:0}: Error finding container ff14a7d361472b00ed67681698a55f587a5e5ae74bf5004aab1334aeac1f00a4: Status 404 returned error can't find the container with id ff14a7d361472b00ed67681698a55f587a5e5ae74bf5004aab1334aeac1f00a4 Jul 16 11:49:37.760112 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.760074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" event={"ID":"0acfeab3-7020-4791-b3e9-da6d0785fe5a","Type":"ContainerStarted","Data":"c3cad868553fb37b106d00cfaa9ceddcdcf8bc0da315020ceabcbd8d914b3a44"} Jul 16 11:49:37.760309 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.760127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" event={"ID":"0acfeab3-7020-4791-b3e9-da6d0785fe5a","Type":"ContainerStarted","Data":"ff14a7d361472b00ed67681698a55f587a5e5ae74bf5004aab1334aeac1f00a4"} Jul 16 11:49:37.760309 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.760210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:37.775725 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:37.775673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" podStartSLOduration=0.77565593 podStartE2EDuration="775.65593ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 11:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 11:49:37.774570269 +0000 UTC m=+2461.540703406" watchObservedRunningTime="2026-07-16 11:49:37.77565593 +0000 UTC m=+2461.541789070" Jul 16 11:49:38.723800 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:38.723771 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c7665bd7d-n5ncj_35c33c4f-7568-4a6d-9047-00f7f37d0117/console/0.log" Jul 16 11:49:38.756850 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:38.756820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-c6vd6_b7cd9313-0cd3-42e5-b793-5adeddf848d0/download-server/0.log" Jul 16 11:49:40.542139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:40.542114 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wr645_048fad95-c706-4a41-9053-6f909977047a/dns/0.log" Jul 16 11:49:40.566921 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:40.566897 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wr645_048fad95-c706-4a41-9053-6f909977047a/kube-rbac-proxy/0.log" Jul 16 11:49:40.652274 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:40.652244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d6gc4_78778387-11b8-464f-8f4f-b791c7e46005/dns-node-resolver/0.log" Jul 16 11:49:41.212623 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:41.212588 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c7989975f-hxnhd_19799a0e-030d-4e24-961a-26122343409d/registry/0.log" Jul 16 11:49:41.263161 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:41.263132 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-js4ck_269c6854-9df1-4fb2-a189-c2d56701199e/node-ca/0.log" Jul 16 11:49:42.498787 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:42.498759 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79cdcbc7df-w8bzg_4e0f2523-eb5b-4fa4-a7df-769a9c01aae5/kube-auth-proxy/0.log" Jul 16 11:49:43.415139 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:43.415094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgjd4_edac3d95-d82e-45ae-8063-fe14ecc387c5/serve-healthcheck-canary/0.log" Jul 16 11:49:43.774052 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:43.773980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8p4bp/perf-node-gather-daemonset-9klcb" Jul 16 11:49:44.023565 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:44.023538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms5rw_46d57d98-2153-4a95-825e-53650d2e7435/kube-rbac-proxy/0.log" Jul 16 11:49:44.044334 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:44.044261 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms5rw_46d57d98-2153-4a95-825e-53650d2e7435/exporter/0.log" Jul 16 11:49:44.065497 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:44.065470 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms5rw_46d57d98-2153-4a95-825e-53650d2e7435/extractor/0.log" Jul 16 11:49:47.184878 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:47.184844 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8c4d5c89-drz7j_829b6646-37b1-48b7-9806-870dd3a62cd6/manager/0.log" Jul 16 11:49:47.203009 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:47.202979 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8c4d5c89-drz7j_829b6646-37b1-48b7-9806-870dd3a62cd6/copy-manifests/0.log" Jul 16 11:49:48.541968 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:48.541935 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-796df6bb6-8mbsw_da6efd8b-3ef2-4346-808a-f6216d0ac660/manager/0.log" Jul 16 11:49:48.664428 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:48.664328 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79cdcbc7df-w8bzg_4e0f2523-eb5b-4fa4-a7df-769a9c01aae5/kube-auth-proxy/0.log" Jul 16 11:49:50.154765 ip-10-0-130-16 kubenswrapper[2580]: I0716 11:49:50.154737 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79cdcbc7df-w8bzg_4e0f2523-eb5b-4fa4-a7df-769a9c01aae5/kube-auth-proxy/0.log"