Jul 06 17:51:52.343229 ip-10-0-129-13 systemd[1]: Starting Kubernetes Kubelet... Jul 06 17:51:52.795457 ip-10-0-129-13 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 17:51:52.795457 ip-10-0-129-13 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 17:51:52.795457 ip-10-0-129-13 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 17:51:52.795457 ip-10-0-129-13 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 17:51:52.795457 ip-10-0-129-13 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 17:51:52.797986 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.797862 2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 17:51:52.802768 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802748 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.802768 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802765 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.802768 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802770 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.802768 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802790 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802800 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802806 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802810 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802815 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802819 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802823 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802827 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802831 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802835 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802839 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802843 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802847 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802852 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802856 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802863 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802869 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802874 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802879 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.803011 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802884 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802888 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802892 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802897 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802902 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802907 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802913 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802918 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802923 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802929 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802934 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802938 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802944 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802949 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802955 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802961 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802966 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802971 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802977 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.803841 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802983 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802990 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802995 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.802999 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803004 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803008 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803013 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803017 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803021 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803025 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803030 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803034 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803039 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803043 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803048 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803052 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803057 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803061 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803073 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803079 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.804719 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803084 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803089 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803094 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803098 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803103 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803108 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803113 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803117 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803123 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803128 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803134 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803138 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803143 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803149 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803154 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803159 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803164 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803168 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803173 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803178 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.805618 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803183 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803187 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803192 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803196 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803201 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803205 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803210 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803215 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803219 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803224 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803229 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803233 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803237 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803242 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803246 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803250 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803255 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.803259 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804521 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804531 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.806515 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804538 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804544 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804548 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804552 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804557 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804562 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804569 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804574 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804579 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804583 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804587 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804592 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804596 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804604 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804610 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804615 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804620 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804625 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804630 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.807466 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804634 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804638 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804642 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804647 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804652 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804657 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804661 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804666 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804670 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804675 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804679 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804683 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804688 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804693 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804697 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804703 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804707 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804712 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804716 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804720 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.808215 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804725 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804732 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804738 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804742 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804746 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804751 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804756 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804760 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804764 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804769 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804797 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804804 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804808 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804813 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804818 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804824 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804828 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804833 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804838 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804843 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.808732 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804847 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804851 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804856 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804861 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804865 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804870 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804874 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804879 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804884 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804889 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804895 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804900 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804905 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804909 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804914 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804920 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804924 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804928 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804933 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804937 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.809342 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804942 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804946 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804951 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804955 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804959 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804964 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804968 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804972 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804977 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804981 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804986 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804992 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.804999 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805003 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805008 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805014 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805018 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805023 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805746 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805756 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.809945 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805762 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805768 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805791 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805798 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805802 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805807 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805812 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805816 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805821 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805825 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805829 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805835 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805839 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805844 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805849 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805854 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805858 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805863 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805867 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805872 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.810528 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805876 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805881 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805885 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805889 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805894 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805899 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805903 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805910 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805916 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805922 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805927 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805932 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805936 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805946 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805951 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805956 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805960 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805964 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805968 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805973 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.811260 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805977 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805982 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805986 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805990 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.805995 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806000 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806008 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806014 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806018 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806024 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806029 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806034 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806039 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806043 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806047 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806052 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806056 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806060 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806065 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.811798 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806074 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806079 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806083 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806088 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806092 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806096 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806101 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806120 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806127 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806131 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806136 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806140 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806145 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806152 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806156 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806161 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806165 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806170 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806175 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806180 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.812381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806184 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806188 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806193 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806197 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806201 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806206 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806210 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806215 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806219 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806223 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806227 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806232 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806238 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806243 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806247 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806252 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806256 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.806261 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806375 2587 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806386 2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806398 2587 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 17:51:52.812926 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806405 2587 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806412 2587 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806417 2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806424 2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806431 2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806437 2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806442 2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806448 2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806454 2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806459 2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806464 2587 flags.go:64] FLAG: --cgroup-root="" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806469 2587 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806474 2587 flags.go:64] FLAG: --client-ca-file="" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806479 2587 flags.go:64] FLAG: --cloud-provider="external" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806485 2587 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806492 2587 flags.go:64] FLAG: --cluster-domain="" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806497 2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806502 2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806507 2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806512 2587 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806519 2587 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806523 2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806528 2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 17:51:52.813470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806535 2587 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806541 2587 flags.go:64] FLAG: --contention-profiling="false" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806546 2587 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806551 2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806557 2587 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806562 2587 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806569 2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806575 2587 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806580 2587 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806585 2587 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806590 2587 flags.go:64] FLAG: --enable-server="true" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806596 2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806604 2587 flags.go:64] FLAG: --event-burst="100" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806609 2587 flags.go:64] FLAG: --event-qps="50" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806614 2587 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806619 2587 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806625 2587 flags.go:64] FLAG: --eviction-hard="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806631 2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806636 2587 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806641 2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806647 2587 flags.go:64] FLAG: --eviction-soft="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806652 2587 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806657 2587 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806662 2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806667 2587 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 17:51:52.814069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806671 2587 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806676 2587 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806681 2587 flags.go:64] FLAG: --feature-gates="" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806688 2587 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806693 2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806698 2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806704 2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806711 2587 flags.go:64] FLAG: --healthz-port="10248" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806716 2587 flags.go:64] FLAG: --help="false" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806721 2587 flags.go:64] FLAG: --hostname-override="ip-10-0-129-13.ec2.internal" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806727 2587 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806732 2587 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806737 2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806743 2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806749 2587 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806754 2587 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806759 2587 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806764 2587 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806770 2587 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806791 2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806796 2587 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806801 2587 flags.go:64] FLAG: --kube-reserved="" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806807 2587 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806812 2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 17:51:52.814758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806817 2587 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806822 2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806827 2587 flags.go:64] FLAG: --lock-file="" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806831 2587 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806836 2587 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806842 2587 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806851 2587 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806856 2587 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806861 2587 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806866 2587 flags.go:64] FLAG: --logging-format="text" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806871 2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806877 2587 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806882 2587 flags.go:64] FLAG: --manifest-url="" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806887 2587 flags.go:64] FLAG: --manifest-url-header="" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806894 2587 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806901 2587 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806909 2587 flags.go:64] FLAG: --max-pods="110" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806914 2587 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806919 2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806924 2587 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806929 2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806934 2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806939 2587 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806944 2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806956 2587 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 17:51:52.815426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806961 2587 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806967 2587 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806973 2587 flags.go:64] FLAG: --pod-cidr="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806978 2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806986 2587 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806991 2587 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.806996 2587 flags.go:64] FLAG: --pods-per-core="0" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807001 2587 flags.go:64] FLAG: --port="10250" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807006 2587 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807011 2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01646b154f63a4f1f" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807017 2587 flags.go:64] FLAG: --qos-reserved="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807022 2587 flags.go:64] FLAG: --read-only-port="10255" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807027 2587 flags.go:64] FLAG: --register-node="true" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807032 2587 flags.go:64] FLAG: --register-with-taints="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807038 2587 flags.go:64] FLAG: --registry-burst="10" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807044 2587 flags.go:64] FLAG: --registry-qps="5" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807048 2587 flags.go:64] FLAG: --reserved-cpus="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807053 2587 flags.go:64] FLAG: --reserved-memory="" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807067 2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807073 2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807078 2587 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807083 2587 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807087 2587 flags.go:64] FLAG: --runonce="false" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807094 2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807102 2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 17:51:52.816246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807107 2587 flags.go:64] FLAG: --seccomp-default="false" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807112 2587 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807118 2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807124 2587 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807131 2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807136 2587 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807141 2587 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807147 2587 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807152 2587 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807158 2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807164 2587 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807169 2587 flags.go:64] FLAG: --system-cgroups="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807175 2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807184 2587 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807189 2587 flags.go:64] FLAG: --tls-cert-file="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807194 2587 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807201 2587 flags.go:64] FLAG: --tls-min-version="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807206 2587 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807211 2587 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807216 2587 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807221 2587 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807227 2587 flags.go:64] FLAG: --v="2" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807233 2587 flags.go:64] FLAG: --version="false" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807240 2587 flags.go:64] FLAG: --vmodule="" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807246 2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 17:51:52.817244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.807252 2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807455 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807462 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807467 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807471 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807478 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807485 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807489 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807493 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807498 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807502 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807508 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807512 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807517 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807522 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807526 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807530 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807536 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807541 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807546 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.817931 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807550 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807554 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807559 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807563 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807568 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807572 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807576 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807581 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807585 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807590 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807594 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807599 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807603 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807607 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807612 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807616 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807621 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807627 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807633 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.818441 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807637 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807644 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807651 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807656 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807661 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807666 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807671 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807675 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807680 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807685 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807689 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807694 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807698 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807703 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807708 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807712 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807716 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807720 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807725 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807729 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.818986 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807734 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807738 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807743 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807747 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807751 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807755 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807760 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807764 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807769 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807790 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807797 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807803 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807808 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807812 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807817 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807821 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807826 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807830 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807836 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.819492 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807842 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807846 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807851 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807856 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807860 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807866 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807870 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807875 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807879 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807883 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807888 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807893 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807897 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807902 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807906 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807911 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807915 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807920 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807924 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807929 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.819989 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807936 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.807941 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.809835 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.818612 2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.818630 2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.818651 2587 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.818657 2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818721 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818726 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818730 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818733 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818737 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818740 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818743 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818746 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.820513 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818749 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818752 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818755 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818757 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818760 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818763 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818765 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818768 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818771 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818789 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818794 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818797 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818800 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818803 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818806 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818809 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818813 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818816 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818819 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818821 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.820925 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818824 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818828 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818831 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818834 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818837 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818840 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818843 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818846 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818848 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818851 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818853 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818856 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818859 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818862 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818865 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818867 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818870 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818873 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818876 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818879 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.821440 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818881 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818884 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818888 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818891 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818895 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818899 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818902 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818905 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818907 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818910 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818913 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818915 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818919 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818924 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818927 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818931 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818934 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818937 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818940 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.822010 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818942 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818945 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818948 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818951 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818953 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818956 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818959 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818963 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818966 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818969 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818972 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818974 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818978 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818981 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818983 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818986 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818989 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818992 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818995 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.822494 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.818998 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819001 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819003 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819006 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819009 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819011 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819014 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819017 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819020 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819022 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819025 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819028 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819030 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.819036 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819159 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 17:51:52.823089 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819164 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819167 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819170 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819173 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819176 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819179 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819182 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819184 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819189 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819193 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819196 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819200 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819204 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819207 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819211 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819214 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819217 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819220 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819222 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 17:51:52.823480 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819225 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819227 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819230 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819233 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819236 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819239 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819242 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819244 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819247 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819250 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819253 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819256 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819258 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819261 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819263 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819266 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819269 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819271 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819274 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819276 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 17:51:52.824170 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819279 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819282 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819284 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819287 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819290 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819293 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819296 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819299 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819302 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819304 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819307 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819310 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819312 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819315 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819318 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819320 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819323 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819325 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819328 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819331 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 17:51:52.824823 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819334 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819337 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819341 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819344 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819347 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819350 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819353 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819356 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819358 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819361 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819363 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819366 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819369 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819372 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819375 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819377 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819380 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819382 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819386 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 17:51:52.825563 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819389 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819392 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819394 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819397 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819399 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819402 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819405 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819407 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819410 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819413 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819418 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819421 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819424 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819426 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819429 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819432 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819434 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819437 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819440 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 17:51:52.826222 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:52.819443 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.819448 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.819596 2587 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.821640 2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.823523 2587 server.go:1019] "Starting client certificate rotation" Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.823642 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.823694 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 17:51:52.826893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.826111 2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 17:51:52.828836 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.828809 2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 17:51:52.836889 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.836873 2587 log.go:25] "Validated CRI v1 runtime API" Jul 06 17:51:52.839749 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.839734 2587 log.go:25] "Validated CRI v1 image API" Jul 06 17:51:52.840612 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.840597 2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 17:51:52.843315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.843290 2587 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa26a6c4-9fc6-4ba3-ad57-9c078163c8e8:/dev/nvme0n1p4 ef1b7a98-db7c-4424-94e5-5da60e065490:/dev/nvme0n1p3] Jul 06 17:51:52.843376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.843314 2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 17:51:52.850221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.850088 2587 manager.go:216] Machine: {Timestamp:2026-07-06 17:51:52.84864818 +0000 UTC m=+0.378180533 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099735 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b5ca203f1eb0f188501989c1ef80 SystemUUID:ec27b5ca-203f-1eb0-f188-501989c1ef80 BootID:fa1dbada-7ebb-423e-afe7-9008e6d15feb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:c5:da:38:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:c5:da:38:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:1a:4f:28:3f:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 17:51:52.850221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.850206 2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 17:51:52.850368 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.850299 2587 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 17:51:52.853568 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.853541 2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 17:51:52.853709 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.853572 2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 17:51:52.853905 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.853890 2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 17:51:52.854934 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.854921 2587 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 17:51:52.854976 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.854937 2587 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 17:51:52.854976 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.854945 2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 17:51:52.855549 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.855539 2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 17:51:52.856730 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.856720 2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 17:51:52.860678 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860667 2587 state_mem.go:36] "Initialized new in-memory state store" Jul 06 17:51:52.860810 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860800 2587 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 17:51:52.860848 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860833 2587 kubelet.go:486] "Attempting to sync node with API server" Jul 06 17:51:52.860848 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860843 2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 17:51:52.860902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860863 2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 17:51:52.860902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860873 2587 kubelet.go:398] "Adding apiserver pod source" Jul 06 17:51:52.860902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.860882 2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 17:51:52.861310 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.861297 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 17:51:52.861358 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.861316 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 17:51:52.864882 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.864864 2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 17:51:52.867722 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.867707 2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 17:51:52.867797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.867788 2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 17:51:52.869023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869011 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 17:51:52.869063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869029 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 17:51:52.869063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869037 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 17:51:52.869063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869042 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 17:51:52.869063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869053 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 17:51:52.869063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869059 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869068 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869075 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869085 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869092 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869110 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 17:51:52.869206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869117 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 17:51:52.869930 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869919 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 17:51:52.869930 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.869930 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 17:51:52.872652 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.872634 2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-13.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 17:51:52.872725 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.872662 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 17:51:52.872917 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.872897 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 17:51:52.873588 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.873422 2587 server.go:1268] "Started kubelet" Jul 06 17:51:52.873588 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.873500 2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 17:51:52.873588 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.873517 2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 17:51:52.873747 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.873592 2587 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 17:51:52.874440 ip-10-0-129-13 systemd[1]: Started Kubernetes Kubelet. Jul 06 17:51:52.875186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.875162 2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 17:51:52.876232 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.876217 2587 server.go:310] "Adding debug handlers to kubelet server" Jul 06 17:51:52.879028 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.879005 2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 17:51:52.883361 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.883341 2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 17:51:52.883361 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.883350 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 17:51:52.883606 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.883573 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:52.884026 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884005 2587 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 17:51:52.884026 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884028 2587 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 17:51:52.884183 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884033 2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 17:51:52.884183 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884147 2587 reconstruct.go:97] "Volume reconstruction finished" Jul 06 17:51:52.884183 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884166 2587 reconciler.go:29] "Reconciler: start to sync state" Jul 06 17:51:52.884395 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884375 2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 17:51:52.884395 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884393 2587 factory.go:55] Registering systemd factory Jul 06 17:51:52.884524 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884403 2587 factory.go:223] Registration of the systemd container factory successfully Jul 06 17:51:52.884621 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884605 2587 factory.go:153] Registering CRI-O factory Jul 06 17:51:52.884674 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884626 2587 factory.go:223] Registration of the crio container factory successfully Jul 06 17:51:52.884674 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884651 2587 factory.go:103] Registering Raw factory Jul 06 17:51:52.884674 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.884665 2587 manager.go:1196] Started watching for new ooms in manager Jul 06 17:51:52.885397 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.885369 2587 manager.go:349] Starting recovery of all containers Jul 06 17:51:52.885582 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.885562 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flb6c" Jul 06 17:51:52.890097 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.890061 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 17:51:52.890209 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.890184 2587 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 17:51:52.891283 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.890157 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-13.ec2.internal.18bfc4c183591040 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-13.ec2.internal,UID:ip-10-0-129-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-13.ec2.internal,},FirstTimestamp:2026-07-06 17:51:52.873390144 +0000 UTC m=+0.402922497,LastTimestamp:2026-07-06 17:51:52.873390144 +0000 UTC m=+0.402922497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-13.ec2.internal,}" Jul 06 17:51:52.895281 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.895256 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flb6c" Jul 06 17:51:52.897560 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.897419 2587 manager.go:354] Recovery completed Jul 06 17:51:52.899147 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.899127 2587 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 17:51:52.902265 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.902252 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:52.904054 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904039 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:52.904131 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904067 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:52.904131 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904093 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:52.904732 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904704 2587 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 17:51:52.904805 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904719 2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 17:51:52.904805 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904748 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:52.904805 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.904756 2587 state_mem.go:36] "Initialized new in-memory state store" Jul 06 17:51:52.905544 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.905529 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:52.905588 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.905559 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:52.905588 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.905575 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:52.906044 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.906032 2587 policy_none.go:49] "None policy: Start" Jul 06 17:51:52.906079 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.906048 2587 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 17:51:52.906079 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.906059 2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 17:51:52.906731 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.906659 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-13.ec2.internal.18bfc4c1852cf3a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-13.ec2.internal,UID:ip-10-0-129-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-13.ec2.internal,},FirstTimestamp:2026-07-06 17:51:52.904053667 +0000 UTC m=+0.433586019,LastTimestamp:2026-07-06 17:51:52.904053667 +0000 UTC m=+0.433586019,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-13.ec2.internal,}" Jul 06 17:51:52.907086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.907074 2587 policy_none.go:47] "Start" Jul 06 17:51:52.951443 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.951412 2587 manager.go:337] "Starting Device Plugin manager" Jul 06 17:51:52.951576 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.951460 2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 17:51:52.951576 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.951473 2587 server.go:87] "Starting device plugin registration server" Jul 06 17:51:52.951838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.951824 2587 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 17:51:52.951927 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.951840 2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 17:51:52.951974 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.951929 2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 17:51:52.952031 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.952018 2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 17:51:52.952031 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:52.952030 2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 17:51:52.952762 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.952736 2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 17:51:52.952907 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:52.952794 2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.056243 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.056160 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:53.057573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.057262 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:53.057573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.057295 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:53.057573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.057306 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:53.057573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.057333 2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.067330 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.067308 2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.067432 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.067334 2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-13.ec2.internal\": node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.068503 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.068480 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 17:51:53.069868 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.069843 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 17:51:53.069974 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.069889 2587 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 17:51:53.069974 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.069915 2587 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 17:51:53.069974 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.069956 2587 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 17:51:53.072282 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.072260 2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 17:51:53.106588 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.106549 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.170440 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.170401 2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal"] Jul 06 17:51:53.170541 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.170496 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:53.171625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.171604 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:53.171762 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.171641 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:53.171762 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.171659 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:53.173105 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173091 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:53.173234 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173216 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.173297 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173259 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:53.173906 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173886 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:53.174006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173912 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:53.174006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173923 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:53.174006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173910 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:53.174006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.173977 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:53.174206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.174017 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:53.176286 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.176270 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.176343 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.176300 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 17:51:53.177084 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.177066 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientMemory" Jul 06 17:51:53.177154 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.177094 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 17:51:53.177154 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.177114 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeHasSufficientPID" Jul 06 17:51:53.199587 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.199564 2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-13.ec2.internal\" not found" node="ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.204418 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.204403 2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-13.ec2.internal\" not found" node="ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.207541 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.207526 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.285436 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.285400 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.285436 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.285437 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.285647 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.285462 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecc2a1fd2e095c365b662a2d934cdfee-config\") pod \"kube-apiserver-proxy-ip-10-0-129-13.ec2.internal\" (UID: \"ecc2a1fd2e095c365b662a2d934cdfee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.308755 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.308690 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.386049 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386023 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.386119 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386055 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.386119 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386072 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecc2a1fd2e095c365b662a2d934cdfee-config\") pod \"kube-apiserver-proxy-ip-10-0-129-13.ec2.internal\" (UID: \"ecc2a1fd2e095c365b662a2d934cdfee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.386200 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386132 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecc2a1fd2e095c365b662a2d934cdfee-config\") pod \"kube-apiserver-proxy-ip-10-0-129-13.ec2.internal\" (UID: \"ecc2a1fd2e095c365b662a2d934cdfee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.386200 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386161 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.386200 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.386132 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d81830a4b3d3ea0cb43e2608cbb03aff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal\" (UID: \"d81830a4b3d3ea0cb43e2608cbb03aff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.409564 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.409540 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.503310 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.503276 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.508573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.508558 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:53.509745 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.509728 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.610843 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.610755 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.712149 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.712118 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.812570 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.812537 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:53.822233 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.822211 2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 17:51:53.822363 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.822346 2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 17:51:53.853128 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.853108 2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 17:51:53.898695 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.898638 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 17:46:52 +0000 UTC" deadline="2027-12-11 01:20:17.615850366 +0000 UTC" Jul 06 17:51:53.898695 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:53.898662 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12535h28m23.717190464s" Jul 06 17:51:53.912765 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:53.912743 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:54.013380 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.013347 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:54.108315 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:54.108286 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc2a1fd2e095c365b662a2d934cdfee.slice/crio-29917c365625e613100f1d283a5ca2293afe9379f7fd4628e42ed3cd57e723eb WatchSource:0}: Error finding container 29917c365625e613100f1d283a5ca2293afe9379f7fd4628e42ed3cd57e723eb: Status 404 returned error can't find the container with id 29917c365625e613100f1d283a5ca2293afe9379f7fd4628e42ed3cd57e723eb Jul 06 17:51:54.108882 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:54.108860 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81830a4b3d3ea0cb43e2608cbb03aff.slice/crio-e2898e0510afe70c9fa907785e665dc30e4f6521563981a54afbfa27155aa1a9 WatchSource:0}: Error finding container e2898e0510afe70c9fa907785e665dc30e4f6521563981a54afbfa27155aa1a9: Status 404 returned error can't find the container with id e2898e0510afe70c9fa907785e665dc30e4f6521563981a54afbfa27155aa1a9 Jul 06 17:51:54.113633 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.113609 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:54.113745 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.113683 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 17:51:54.214119 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.214036 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-13.ec2.internal\" not found" Jul 06 17:51:54.300498 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.300469 2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 17:51:54.384238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.384210 2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" Jul 06 17:51:54.398855 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.398832 2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 17:51:54.399901 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.399889 2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" Jul 06 17:51:54.409754 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.409738 2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 17:51:54.438351 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.438329 2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 17:51:54.862483 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.862458 2587 apiserver.go:52] "Watching apiserver" Jul 06 17:51:54.876515 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.876487 2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 17:51:54.877891 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.877391 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-c955d","openshift-network-diagnostics/network-check-target-s85lm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r","openshift-cluster-node-tuning-operator/tuned-hgtcx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal","openshift-multus/multus-2p9dl","openshift-network-operator/iptables-alerter-cdt9b","openshift-ovn-kubernetes/ovnkube-node-hb6qz","kube-system/global-pull-secret-syncer-ms7tt","kube-system/konnectivity-agent-4crmj","kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal","openshift-dns/node-resolver-4lnnp","openshift-image-registry/node-ca-mfbkd","openshift-multus/multus-additional-cni-plugins-928jl"] Jul 06 17:51:54.879434 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.879407 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.880603 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.880579 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.881832 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.881813 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.882279 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.882257 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnths\"" Jul 06 17:51:54.882429 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.882362 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 17:51:54.883212 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.883194 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.883622 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.883603 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.884043 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.884026 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.884265 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.884247 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 17:51:54.884374 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.884351 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 17:51:54.885104 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885085 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.885185 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885128 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 17:51:54.885290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885273 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.885378 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885362 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 17:51:54.885490 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885474 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47kvm\"" Jul 06 17:51:54.885580 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885564 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.885667 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885652 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 17:51:54.885962 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.885946 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:54.886038 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.886009 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:51:54.886094 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886074 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.886320 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886296 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 17:51:54.886395 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886384 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.886660 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886643 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t89xh\"" Jul 06 17:51:54.886875 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886857 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.886937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.886875 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.887129 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.887111 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6lm29\"" Jul 06 17:51:54.887758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.887742 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:54.887831 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.887812 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:51:54.888780 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.888763 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:54.889238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.889089 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.889899 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.889667 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hqfj2\"" Jul 06 17:51:54.889899 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.889696 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 17:51:54.889899 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.889772 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 17:51:54.890140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.890012 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:54.890140 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.890064 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:51:54.890291 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.890165 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.891369 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.891349 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:54.891739 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.891707 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 17:51:54.891821 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.891725 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.892182 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.892164 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.892825 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.892807 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4txcc\"" Jul 06 17:51:54.893275 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.893259 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.893935 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.893917 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 17:51:54.894347 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894315 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mxs8f\"" Jul 06 17:51:54.894451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894387 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovn-node-metrics-cert\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.894451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894416 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.894451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894442 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-modprobe-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894458 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-tuned\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894483 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-netns\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894506 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z9z\" (UniqueName: \"kubernetes.io/projected/7e14a4ba-0d02-4f44-a101-9f92ade15220-kube-api-access-g5z9z\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894528 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-kubernetes\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894549 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-lib-modules\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894565 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-var-lib-kubelet\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894579 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-tmp\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894600 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894630 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-systemd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.894627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894647 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bcc\" (UniqueName: \"kubernetes.io/projected/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-kube-api-access-b2bcc\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894667 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28ed05bb-84b3-4264-b747-e7412603cf1f-host-slash\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894682 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-var-lib-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894690 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894704 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894745 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-registration-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894760 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-sys-fs\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894778 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysconfig\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894798 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-run\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894811 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-sys\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894824 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-env-overrides\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894844 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-kubelet\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894859 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-node-log\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894875 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-log-socket\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894892 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894915 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-netd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894941 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-config\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.895190 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894961 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894977 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28ed05bb-84b3-4264-b747-e7412603cf1f-iptables-alerter-script\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.894997 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-slash\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895013 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-bin\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895040 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-script-lib\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895083 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-device-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895108 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895128 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-conf\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895141 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-host\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895155 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gbbx\" (UniqueName: \"kubernetes.io/projected/051b52de-5f8c-4f51-88fb-0b286ab70161-kube-api-access-9gbbx\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895189 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-etc-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895212 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-socket-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895235 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-systemd\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895259 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/28ed05bb-84b3-4264-b747-e7412603cf1f-kube-api-access-d58k4\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895283 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-systemd-units\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895305 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-ovn\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.896108 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895335 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtbv\" (UniqueName: \"kubernetes.io/projected/798cac93-2f31-4157-b50e-b5281409046b-kube-api-access-lmtbv\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.896966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895359 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.896966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895446 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:54.896966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895862 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v4v74\"" Jul 06 17:51:54.896966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.895974 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 17:51:54.896966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.896441 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 17:51:54.897620 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.897598 2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 17:51:54.898024 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.898005 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 17:51:54.898024 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.898007 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 17:51:54.898452 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.898238 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-22mbk\"" Jul 06 17:51:54.922684 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.922656 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-852pd" Jul 06 17:51:54.928801 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.928781 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-852pd" Jul 06 17:51:54.984849 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.984830 2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 17:51:54.995863 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.995833 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-registration-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.995966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.995875 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-netns\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.995966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.995899 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-dbus\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:54.995966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.995914 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-registration-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.995966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.995922 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-env-overrides\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996153 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996035 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-kubelet\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996153 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996088 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-netd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996153 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996111 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-config\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996153 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996138 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c190a68-a52e-49b3-b6fc-1bbff1065658-konnectivity-ca\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996170 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4nb5\" (UniqueName: \"kubernetes.io/projected/6024ad2c-4426-47fd-a007-44e9da76963d-kube-api-access-x4nb5\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996176 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-netd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996167 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-kubelet\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996213 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-bin\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996249 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-device-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996273 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-conf\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.996325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996300 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996323 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-cni-bin\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996328 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-cnibin\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996382 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-tmp-dir\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996389 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-device-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996412 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-etc-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996437 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-systemd\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996450 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-conf\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996463 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-k8s-cni-cncf-io\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996446 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-etc-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996470 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-env-overrides\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996499 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-bin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996524 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-systemd\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996543 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-hosts-file\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996571 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/28ed05bb-84b3-4264-b747-e7412603cf1f-kube-api-access-d58k4\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996592 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-systemd-units\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996608 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtbv\" (UniqueName: \"kubernetes.io/projected/798cac93-2f31-4157-b50e-b5281409046b-kube-api-access-lmtbv\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.996727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996660 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-config\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996673 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-systemd-units\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996662 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996709 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-system-cni-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996749 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-os-release\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996767 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996793 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-modprobe-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996809 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-cnibin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996825 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddkj\" (UniqueName: \"kubernetes.io/projected/efb406b4-8fd5-44d2-a657-6f6691c098a2-kube-api-access-xddkj\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996860 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-var-lib-kubelet\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996896 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-tmp\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996919 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996929 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-modprobe-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996934 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysctl-d\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996932 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-var-lib-kubelet\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.996939 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bcc\" (UniqueName: \"kubernetes.io/projected/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-kube-api-access-b2bcc\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.997034 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:54.997477 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:54.997112 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:55.497092894 +0000 UTC m=+3.026625245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997138 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-cni-binary-copy\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997190 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28ed05bb-84b3-4264-b747-e7412603cf1f-host-slash\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997217 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-var-lib-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997243 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997262 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28ed05bb-84b3-4264-b747-e7412603cf1f-host-slash\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997269 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-sys-fs\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997242 2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997278 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-var-lib-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997301 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysconfig\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997309 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-openvswitch\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997341 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-run\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997361 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-sys\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997361 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-sys-fs\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997367 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-sysconfig\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997384 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997407 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-run\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997414 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-node-log\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997443 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-log-socket\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997417 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-sys\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997449 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-node-log\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997467 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997495 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997518 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997520 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997510 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-log-socket\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997565 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-host\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997571 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997618 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-system-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997621 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997644 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-os-release\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997670 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28ed05bb-84b3-4264-b747-e7412603cf1f-iptables-alerter-script\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997684 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-host\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997701 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-slash\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997744 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-script-lib\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.998804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997770 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-slash\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997773 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-kubelet\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997809 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-conf-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997841 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-multus-daemon-config\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997867 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-etc-kubernetes\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997891 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9gl\" (UniqueName: \"kubernetes.io/projected/d0316b37-42d5-4ac9-86a7-9baecdd16214-kube-api-access-vk9gl\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997919 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gbbx\" (UniqueName: \"kubernetes.io/projected/051b52de-5f8c-4f51-88fb-0b286ab70161-kube-api-access-9gbbx\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997945 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-socket-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997968 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efb406b4-8fd5-44d2-a657-6f6691c098a2-serviceca\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.997992 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998015 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998039 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-kubelet-config\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998066 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-ovn\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998089 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-multus\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998111 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-hostroot\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998134 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9zq\" (UniqueName: \"kubernetes.io/projected/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-kube-api-access-kc9zq\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:54.999583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998155 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28ed05bb-84b3-4264-b747-e7412603cf1f-iptables-alerter-script\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998161 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovn-node-metrics-cert\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998211 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998238 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-tuned\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998266 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-socket-dir-parent\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998290 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998329 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-netns\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998355 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z9z\" (UniqueName: \"kubernetes.io/projected/7e14a4ba-0d02-4f44-a101-9f92ade15220-kube-api-access-g5z9z\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998357 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovnkube-script-lib\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998381 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-kubernetes\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998405 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-lib-modules\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998426 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-multus-certs\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998440 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998447 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb406b4-8fd5-44d2-a657-6f6691c098a2-host\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998469 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-systemd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998493 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c190a68-a52e-49b3-b6fc-1bbff1065658-agent-certs\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998505 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-host-run-netns\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.000479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998518 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-binary-copy\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998651 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-lib-modules\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998742 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-kubernetes\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998752 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-systemd\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998839 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e14a4ba-0d02-4f44-a101-9f92ade15220-run-ovn\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:54.998873 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cac93-2f31-4157-b50e-b5281409046b-socket-dir\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:55.001379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.001063 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-etc-tuned\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.001681 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.001420 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e14a4ba-0d02-4f44-a101-9f92ade15220-ovn-node-metrics-cert\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.001681 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.001565 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-tmp\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.011731 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.011697 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/28ed05bb-84b3-4264-b747-e7412603cf1f-kube-api-access-d58k4\") pod \"iptables-alerter-cdt9b\" (UID: \"28ed05bb-84b3-4264-b747-e7412603cf1f\") " pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:55.016502 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.016481 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bcc\" (UniqueName: \"kubernetes.io/projected/e7bd801f-b41e-4e9a-934c-260fac6e9b8c-kube-api-access-b2bcc\") pod \"tuned-hgtcx\" (UID: \"e7bd801f-b41e-4e9a-934c-260fac6e9b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.017156 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.017135 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtbv\" (UniqueName: \"kubernetes.io/projected/798cac93-2f31-4157-b50e-b5281409046b-kube-api-access-lmtbv\") pod \"aws-ebs-csi-driver-node-xnh8r\" (UID: \"798cac93-2f31-4157-b50e-b5281409046b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:55.017525 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.017483 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gbbx\" (UniqueName: \"kubernetes.io/projected/051b52de-5f8c-4f51-88fb-0b286ab70161-kube-api-access-9gbbx\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:55.017525 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.017523 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z9z\" (UniqueName: \"kubernetes.io/projected/7e14a4ba-0d02-4f44-a101-9f92ade15220-kube-api-access-g5z9z\") pod \"ovnkube-node-hb6qz\" (UID: \"7e14a4ba-0d02-4f44-a101-9f92ade15220\") " pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.074437 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.074392 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" event={"ID":"ecc2a1fd2e095c365b662a2d934cdfee","Type":"ContainerStarted","Data":"29917c365625e613100f1d283a5ca2293afe9379f7fd4628e42ed3cd57e723eb"} Jul 06 17:51:55.075334 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.075312 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" event={"ID":"d81830a4b3d3ea0cb43e2608cbb03aff","Type":"ContainerStarted","Data":"e2898e0510afe70c9fa907785e665dc30e4f6521563981a54afbfa27155aa1a9"} Jul 06 17:51:55.099411 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099382 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-binary-copy\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.099411 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099412 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-netns\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099431 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-dbus\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099458 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c190a68-a52e-49b3-b6fc-1bbff1065658-konnectivity-ca\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099496 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4nb5\" (UniqueName: \"kubernetes.io/projected/6024ad2c-4426-47fd-a007-44e9da76963d-kube-api-access-x4nb5\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099521 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099545 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-cnibin\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099566 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-tmp-dir\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.099593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099593 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-k8s-cni-cncf-io\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099522 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-netns\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099618 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-bin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099649 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-hosts-file\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099676 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-system-cni-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099730 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-os-release\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099758 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099782 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-cnibin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099823 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddkj\" (UniqueName: \"kubernetes.io/projected/efb406b4-8fd5-44d2-a657-6f6691c098a2-kube-api-access-xddkj\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099863 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-cni-binary-copy\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.099904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099875 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-dbus\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099921 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099952 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-cnibin\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100033 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-binary-copy\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100042 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-os-release\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.099898 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100107 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-system-cni-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100114 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-system-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100140 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-os-release\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100170 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-kubelet\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100193 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-conf-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100219 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-multus-daemon-config\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100244 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-etc-kubernetes\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100269 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9gl\" (UniqueName: \"kubernetes.io/projected/d0316b37-42d5-4ac9-86a7-9baecdd16214-kube-api-access-vk9gl\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.100364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100320 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-tmp-dir\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100400 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-system-cni-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100417 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-k8s-cni-cncf-io\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.100193 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100450 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-cnibin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100491 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-kubelet\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.100502 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:55.600483649 +0000 UTC m=+3.130015994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100524 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c190a68-a52e-49b3-b6fc-1bbff1065658-konnectivity-ca\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100539 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-conf-dir\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100545 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-os-release\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100186 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-hosts-file\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100588 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-etc-kubernetes\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100731 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-bin\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100784 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efb406b4-8fd5-44d2-a657-6f6691c098a2-serviceca\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100811 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-cni-binary-copy\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100829 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100855 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100905 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-kubelet-config\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.101023 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100935 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-multus\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100958 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-hostroot\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.100981 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9zq\" (UniqueName: \"kubernetes.io/projected/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-kube-api-access-kc9zq\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101009 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-socket-dir-parent\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101015 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0316b37-42d5-4ac9-86a7-9baecdd16214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101034 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101066 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-multus-certs\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101071 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-hostroot\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101087 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb406b4-8fd5-44d2-a657-6f6691c098a2-host\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101111 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c190a68-a52e-49b3-b6fc-1bbff1065658-agent-certs\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101113 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6024ad2c-4426-47fd-a007-44e9da76963d-multus-daemon-config\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101123 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df8e680c-48eb-41b1-a753-ac4688d2ef96-kubelet-config\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101166 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-var-lib-cni-multus\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101221 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-multus-socket-dir-parent\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101222 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efb406b4-8fd5-44d2-a657-6f6691c098a2-serviceca\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101261 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6024ad2c-4426-47fd-a007-44e9da76963d-host-run-multus-certs\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101277 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb406b4-8fd5-44d2-a657-6f6691c098a2-host\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101412 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.101888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.101689 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0316b37-42d5-4ac9-86a7-9baecdd16214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.104667 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.104647 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c190a68-a52e-49b3-b6fc-1bbff1065658-agent-certs\") pod \"konnectivity-agent-4crmj\" (UID: \"3c190a68-a52e-49b3-b6fc-1bbff1065658\") " pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.118744 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.118677 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:51:55.118744 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.118700 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:51:55.118744 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.118730 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:55.118933 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.118800 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:55.618783472 +0000 UTC m=+3.148315814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:55.119451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.119426 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4nb5\" (UniqueName: \"kubernetes.io/projected/6024ad2c-4426-47fd-a007-44e9da76963d-kube-api-access-x4nb5\") pod \"multus-2p9dl\" (UID: \"6024ad2c-4426-47fd-a007-44e9da76963d\") " pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.119726 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.119695 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9gl\" (UniqueName: \"kubernetes.io/projected/d0316b37-42d5-4ac9-86a7-9baecdd16214-kube-api-access-vk9gl\") pod \"multus-additional-cni-plugins-928jl\" (UID: \"d0316b37-42d5-4ac9-86a7-9baecdd16214\") " pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.121491 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.121473 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9zq\" (UniqueName: \"kubernetes.io/projected/59cb332e-0f1b-4204-9c67-4df28a5ddbc3-kube-api-access-kc9zq\") pod \"node-resolver-4lnnp\" (UID: \"59cb332e-0f1b-4204-9c67-4df28a5ddbc3\") " pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.123054 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.123038 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddkj\" (UniqueName: \"kubernetes.io/projected/efb406b4-8fd5-44d2-a657-6f6691c098a2-kube-api-access-xddkj\") pod \"node-ca-mfbkd\" (UID: \"efb406b4-8fd5-44d2-a657-6f6691c098a2\") " pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.196125 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.196100 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cdt9b" Jul 06 17:51:55.205242 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.205224 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:51:55.214886 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.214864 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" Jul 06 17:51:55.220898 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.220883 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" Jul 06 17:51:55.227893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.227871 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2p9dl" Jul 06 17:51:55.235524 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.235498 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfbkd" Jul 06 17:51:55.244049 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.244026 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:51:55.250203 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.250181 2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 17:51:55.251242 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.251225 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-928jl" Jul 06 17:51:55.256843 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.256821 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lnnp" Jul 06 17:51:55.503886 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.503818 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:55.504028 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.503928 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:55.504028 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.503988 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:56.503969536 +0000 UTC m=+4.033501875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:55.604496 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.604451 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:55.604676 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.604603 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:55.604676 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.604655 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:56.604641766 +0000 UTC m=+4.134174104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:55.705477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.705435 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:55.705675 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.705628 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:51:55.705675 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.705652 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:51:55.705675 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.705665 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:55.705880 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:55.705749 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:56.70572802 +0000 UTC m=+4.235260376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:55.931345 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.931288 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 17:46:54 +0000 UTC" deadline="2028-02-20 08:17:17.155595901 +0000 UTC" Jul 06 17:51:55.931345 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:55.931345 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14246h25m21.224256122s" Jul 06 17:51:56.021315 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.021285 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cb332e_0f1b_4204_9c67_4df28a5ddbc3.slice/crio-11c18655e9e0245a24ad01e0cd8df539241de17ef8d1aa685b5f56a2d911776d WatchSource:0}: Error finding container 11c18655e9e0245a24ad01e0cd8df539241de17ef8d1aa685b5f56a2d911776d: Status 404 returned error can't find the container with id 11c18655e9e0245a24ad01e0cd8df539241de17ef8d1aa685b5f56a2d911776d Jul 06 17:51:56.026029 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.026007 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bd801f_b41e_4e9a_934c_260fac6e9b8c.slice/crio-0c09e1305bb75a5420f28e85e0fc54875decb6c65e69297e849403a029acd284 WatchSource:0}: Error finding container 0c09e1305bb75a5420f28e85e0fc54875decb6c65e69297e849403a029acd284: Status 404 returned error can't find the container with id 0c09e1305bb75a5420f28e85e0fc54875decb6c65e69297e849403a029acd284 Jul 06 17:51:56.027259 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.027238 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798cac93_2f31_4157_b50e_b5281409046b.slice/crio-d8bba9860279585c038b2e6ec5d966887d2294aa2e39b35df0f8d7e8ca6d5020 WatchSource:0}: Error finding container d8bba9860279585c038b2e6ec5d966887d2294aa2e39b35df0f8d7e8ca6d5020: Status 404 returned error can't find the container with id d8bba9860279585c038b2e6ec5d966887d2294aa2e39b35df0f8d7e8ca6d5020 Jul 06 17:51:56.027930 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.027910 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c190a68_a52e_49b3_b6fc_1bbff1065658.slice/crio-9e36d330b0da97a804b6b35d169402ae125055571a63954fc722ed24a33def36 WatchSource:0}: Error finding container 9e36d330b0da97a804b6b35d169402ae125055571a63954fc722ed24a33def36: Status 404 returned error can't find the container with id 9e36d330b0da97a804b6b35d169402ae125055571a63954fc722ed24a33def36 Jul 06 17:51:56.029600 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.029269 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e14a4ba_0d02_4f44_a101_9f92ade15220.slice/crio-0910f010bfb551cae5fdcf6bba68f6ac93c4724211f78aee2a7cbbd48d6bec47 WatchSource:0}: Error finding container 0910f010bfb551cae5fdcf6bba68f6ac93c4724211f78aee2a7cbbd48d6bec47: Status 404 returned error can't find the container with id 0910f010bfb551cae5fdcf6bba68f6ac93c4724211f78aee2a7cbbd48d6bec47 Jul 06 17:51:56.030659 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:51:56.030596 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0316b37_42d5_4ac9_86a7_9baecdd16214.slice/crio-103b3400e0085cdda131cc42fe474d919067145c8db87b08715bb5268b1c612c WatchSource:0}: Error finding container 103b3400e0085cdda131cc42fe474d919067145c8db87b08715bb5268b1c612c: Status 404 returned error can't find the container with id 103b3400e0085cdda131cc42fe474d919067145c8db87b08715bb5268b1c612c Jul 06 17:51:56.070743 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.070616 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:56.070838 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.070821 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:51:56.078329 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.078296 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfbkd" event={"ID":"efb406b4-8fd5-44d2-a657-6f6691c098a2","Type":"ContainerStarted","Data":"f459b4127c8809dbc3938550d1f76369ff6925a266916a0172fe1fd4eeff3353"} Jul 06 17:51:56.080080 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.080054 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2p9dl" event={"ID":"6024ad2c-4426-47fd-a007-44e9da76963d","Type":"ContainerStarted","Data":"42b82d91ec857bec12c3e7f34a526c953b4ca3a91ae6df7ee9feb7f0eedd6a7c"} Jul 06 17:51:56.082003 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.081978 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cdt9b" event={"ID":"28ed05bb-84b3-4264-b747-e7412603cf1f","Type":"ContainerStarted","Data":"dd1a745b33f6f4bbe926de476ed29ff5e8452c31dd95a0d696d65c21942d0291"} Jul 06 17:51:56.083072 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.083047 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"0910f010bfb551cae5fdcf6bba68f6ac93c4724211f78aee2a7cbbd48d6bec47"} Jul 06 17:51:56.084130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.084109 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" event={"ID":"798cac93-2f31-4157-b50e-b5281409046b","Type":"ContainerStarted","Data":"d8bba9860279585c038b2e6ec5d966887d2294aa2e39b35df0f8d7e8ca6d5020"} Jul 06 17:51:56.085212 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.085192 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lnnp" event={"ID":"59cb332e-0f1b-4204-9c67-4df28a5ddbc3","Type":"ContainerStarted","Data":"11c18655e9e0245a24ad01e0cd8df539241de17ef8d1aa685b5f56a2d911776d"} Jul 06 17:51:56.086381 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.086361 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerStarted","Data":"103b3400e0085cdda131cc42fe474d919067145c8db87b08715bb5268b1c612c"} Jul 06 17:51:56.087478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.087458 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4crmj" event={"ID":"3c190a68-a52e-49b3-b6fc-1bbff1065658","Type":"ContainerStarted","Data":"9e36d330b0da97a804b6b35d169402ae125055571a63954fc722ed24a33def36"} Jul 06 17:51:56.088496 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.088477 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" event={"ID":"e7bd801f-b41e-4e9a-934c-260fac6e9b8c","Type":"ContainerStarted","Data":"0c09e1305bb75a5420f28e85e0fc54875decb6c65e69297e849403a029acd284"} Jul 06 17:51:56.511401 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.511320 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:56.511570 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.511522 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:56.511624 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.511591 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:58.511572614 +0000 UTC m=+6.041104958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:56.612562 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.612497 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:56.612760 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.612688 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:56.612830 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.612787 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:58.612768439 +0000 UTC m=+6.142300784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:56.713944 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.713874 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:56.714121 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.714053 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:51:56.714121 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.714075 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:51:56.714234 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.714135 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:56.714234 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:56.714196 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:51:58.714177758 +0000 UTC m=+6.243710110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:56.932308 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.932265 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 17:46:54 +0000 UTC" deadline="2027-12-03 11:44:55.845161073 +0000 UTC" Jul 06 17:51:56.932308 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:56.932305 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12353h52m58.912859913s" Jul 06 17:51:57.073134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.073103 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:57.073313 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:57.073231 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:51:57.073668 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.073650 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:57.073833 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:57.073813 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:51:57.098392 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.098357 2587 generic.go:356] "Generic (PLEG): container finished" podID="d81830a4b3d3ea0cb43e2608cbb03aff" containerID="6730fe865284cd31c5c31f8c508ac42f09451261f5755b3b2f3d105f79c685f9" exitCode=0 Jul 06 17:51:57.098534 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.098453 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" event={"ID":"d81830a4b3d3ea0cb43e2608cbb03aff","Type":"ContainerDied","Data":"6730fe865284cd31c5c31f8c508ac42f09451261f5755b3b2f3d105f79c685f9"} Jul 06 17:51:57.124455 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.124422 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" event={"ID":"ecc2a1fd2e095c365b662a2d934cdfee","Type":"ContainerStarted","Data":"54f01a7ad20303a804fcc2d0352bb00738dd299c73419d3dccbda39f79bd8253"} Jul 06 17:51:57.143976 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:57.143924 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-13.ec2.internal" podStartSLOduration=3.143903969 podStartE2EDuration="3.143903969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:51:57.143041934 +0000 UTC m=+4.672574295" watchObservedRunningTime="2026-07-06 17:51:57.143903969 +0000 UTC m=+4.673436331" Jul 06 17:51:58.070328 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:58.070295 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:58.070823 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.070432 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:51:58.142279 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:58.142245 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" event={"ID":"d81830a4b3d3ea0cb43e2608cbb03aff","Type":"ContainerStarted","Data":"fda5eefeed6e789832c0b10e7c89d97ec054ee8c51d1d3952869d9a5372f1077"} Jul 06 17:51:58.534007 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:58.533914 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:58.534207 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.534055 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:58.534207 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.534120 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:02.534101644 +0000 UTC m=+10.063633986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:51:58.635250 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:58.635213 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:51:58.635440 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.635398 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:58.635547 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.635460 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:02.635441664 +0000 UTC m=+10.164974004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:51:58.736373 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:58.736336 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:58.736559 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.736544 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:51:58.736628 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.736564 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:51:58.736628 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.736578 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:58.736739 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:58.736636 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:02.736618573 +0000 UTC m=+10.266150929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:51:59.073084 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:59.073046 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:51:59.073551 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:59.073167 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:51:59.073641 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:51:59.073557 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:51:59.073694 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:51:59.073665 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:00.070311 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:00.070273 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:00.070494 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:00.070419 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:01.071162 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:01.071080 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:01.071640 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:01.071233 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:01.071640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:01.071285 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:01.071640 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:01.071440 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:02.070534 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:02.070493 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:02.070733 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.070624 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:02.570197 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:02.570157 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:02.570673 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.570348 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:02.570673 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.570441 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:10.570418544 +0000 UTC m=+18.099950884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:02.671275 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:02.671235 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:02.671443 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.671408 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:02.671518 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.671488 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:10.671468272 +0000 UTC m=+18.201000612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:02.771703 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:02.771619 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:02.771870 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.771811 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:52:02.771870 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.771839 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:52:02.771870 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.771852 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:02.772050 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:02.771917 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:10.77189656 +0000 UTC m=+18.301428901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:03.071957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:03.071330 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:03.071957 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:03.071454 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:03.071957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:03.071829 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:03.071957 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:03.071918 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:04.071012 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:04.070510 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:04.071012 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:04.070650 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:05.070244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:05.070211 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:05.070445 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:05.070209 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:05.070445 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:05.070360 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:05.070566 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:05.070515 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:06.070609 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:06.070585 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:06.070919 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:06.070688 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:07.070174 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:07.070144 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:07.070373 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:07.070149 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:07.070373 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:07.070263 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:07.070373 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:07.070350 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:08.070620 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:08.070432 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:08.070620 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:08.070559 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:09.070293 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:09.070263 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:09.070482 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:09.070263 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:09.070482 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:09.070379 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:09.070592 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:09.070492 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:10.070586 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:10.070553 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:10.071001 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.070659 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:10.632543 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:10.632499 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:10.632843 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.632675 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:10.632843 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.632771 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:26.632748124 +0000 UTC m=+34.162280479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:10.733067 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:10.733031 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:10.733249 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.733187 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:10.733309 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.733260 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:26.733245143 +0000 UTC m=+34.262777485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:10.834295 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:10.834256 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:10.834458 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.834396 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:52:10.834458 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.834422 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:52:10.834458 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.834438 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:10.834597 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:10.834501 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:26.834485924 +0000 UTC m=+34.364018285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:11.071295 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:11.071251 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:11.071769 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:11.071413 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:11.071769 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:11.071465 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:11.071769 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:11.071661 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:12.070505 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:12.070468 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:12.070696 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:12.070593 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:13.070999 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:13.070967 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:13.071461 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:13.071080 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:13.071461 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:13.071143 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:13.071461 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:13.071243 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:14.070962 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:14.070927 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:14.071175 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:14.071031 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:15.070450 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.070265 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:15.070611 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.070348 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:15.070611 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:15.070524 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:15.070692 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:15.070634 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:15.179418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.179175 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" event={"ID":"e7bd801f-b41e-4e9a-934c-260fac6e9b8c","Type":"ContainerStarted","Data":"789a3f58578025589d6b033df1a3e9a6f85b533558799d6f8217e3e5b507e0e8"} Jul 06 17:52:15.181078 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.181049 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfbkd" event={"ID":"efb406b4-8fd5-44d2-a657-6f6691c098a2","Type":"ContainerStarted","Data":"16bc95a2374f7205e6eb8555386c64ed60308df27b2ae96dffc249a414d9c267"} Jul 06 17:52:15.182698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.182676 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2p9dl" event={"ID":"6024ad2c-4426-47fd-a007-44e9da76963d","Type":"ContainerStarted","Data":"c17c1e0774645fbaa4bc44cf516204706bb8853768bcf22a9d633822435f8d57"} Jul 06 17:52:15.185952 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.185899 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"8707fd23dd7bf42ac6b1823676df07939c7687e05f9c41f70ce3793f90cb0dc6"} Jul 06 17:52:15.186055 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.186006 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"2506ec82f23f05f47ad19a1519532712ec3027f3a461ac8d3144247b702c71d0"} Jul 06 17:52:15.186055 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.186052 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"15d65fdcc4bad356a8b9ae615399f0a8edf4221c625fadbb1a2574a330ce4a02"} Jul 06 17:52:15.186154 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.186097 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"c11f798d02a65e8de71f621ebb52e320b5757e0cc3d04daa4ae0370a12e40585"} Jul 06 17:52:15.186154 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.186145 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"4e502a3f837c4ab15c990ff82b3cb6f6138d8e4dfd0a66ec6205ae1c2a475d8c"} Jul 06 17:52:15.186247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.186193 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"2a49ffef06ee2624bc390e97711bdd054fa8ad61f7c419bc9137a8bc1f9f78d4"} Jul 06 17:52:15.187441 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.187420 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" event={"ID":"798cac93-2f31-4157-b50e-b5281409046b","Type":"ContainerStarted","Data":"af1425bd092777db332bc81556565d95f2c779afa89c1ff08ec9589a298c7a60"} Jul 06 17:52:15.188856 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.188826 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lnnp" event={"ID":"59cb332e-0f1b-4204-9c67-4df28a5ddbc3","Type":"ContainerStarted","Data":"6fbcc5bcfe3bf2f5b524d8925eabaabf27ab4b4e960d776989bf6ae73652ad37"} Jul 06 17:52:15.190587 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.190566 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="ed5fe22020c2fe0d7aa7606e13aad2349d99020fbaf3216b2e0fb42b2f2c989c" exitCode=0 Jul 06 17:52:15.190687 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.190645 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"ed5fe22020c2fe0d7aa7606e13aad2349d99020fbaf3216b2e0fb42b2f2c989c"} Jul 06 17:52:15.192152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.192123 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4crmj" event={"ID":"3c190a68-a52e-49b3-b6fc-1bbff1065658","Type":"ContainerStarted","Data":"9cc0d1978f21ef141f76d628cfa4cf7978a318f679c04294834c9872dfcd04d2"} Jul 06 17:52:15.198685 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.198644 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hgtcx" podStartSLOduration=3.902664535 podStartE2EDuration="22.198631857s" totalImagesPullingTime="18.295967322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:15.198064313 +0000 UTC m=+22.727596673" watchObservedRunningTime="2026-07-06 17:52:15.198631857 +0000 UTC m=+22.728164217" Jul 06 17:52:15.198805 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.198767 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-13.ec2.internal" podStartSLOduration=21.198761483 podStartE2EDuration="21.198761483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:51:58.165572422 +0000 UTC m=+5.695104785" watchObservedRunningTime="2026-07-06 17:52:15.198761483 +0000 UTC m=+22.728293842" Jul 06 17:52:15.232357 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.229750 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4lnnp" podStartSLOduration=3.932988145 podStartE2EDuration="22.229730334s" totalImagesPullingTime="18.296742189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:15.22948951 +0000 UTC m=+22.759021870" watchObservedRunningTime="2026-07-06 17:52:15.229730334 +0000 UTC m=+22.759262686" Jul 06 17:52:15.232357 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.229876 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mfbkd" podStartSLOduration=12.187145294 podStartE2EDuration="22.229870778s" totalImagesPullingTime="10.042725484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:15.213091816 +0000 UTC m=+22.742624175" watchObservedRunningTime="2026-07-06 17:52:15.229870778 +0000 UTC m=+22.759403139" Jul 06 17:52:15.251479 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.251419 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2p9dl" podStartSLOduration=3.910683379 podStartE2EDuration="22.251398491s" totalImagesPullingTime="18.340715112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:15.250825613 +0000 UTC m=+22.780357974" watchObservedRunningTime="2026-07-06 17:52:15.251398491 +0000 UTC m=+22.780930852" Jul 06 17:52:15.271930 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.271885 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4crmj" podStartSLOduration=8.290108094 podStartE2EDuration="22.271870408s" totalImagesPullingTime="13.981762314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:15.271261096 +0000 UTC m=+22.800793456" watchObservedRunningTime="2026-07-06 17:52:15.271870408 +0000 UTC m=+22.801402768" Jul 06 17:52:15.298472 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.298444 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:15.381610 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.381581 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:15.446905 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:15.446819 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:16.067813 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.067787 2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 17:52:16.070889 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.070870 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:16.071007 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:16.070989 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:16.196492 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.196401 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" event={"ID":"798cac93-2f31-4157-b50e-b5281409046b","Type":"ContainerStarted","Data":"25588cc81c2ed78456b4569f7ceaf35d7519f0ce8855ecc54e8ea3c65f2d44e6"} Jul 06 17:52:16.198067 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.198023 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cdt9b" event={"ID":"28ed05bb-84b3-4264-b747-e7412603cf1f","Type":"ContainerStarted","Data":"f6669127c88c28d986298d3eac7d7d93394c601937eccd1a5119cf4743071ef9"} Jul 06 17:52:16.347069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.347033 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:16.448372 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.448293 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:16.961941 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.961788 2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T17:52:16.067806955Z","UUID":"f8305216-dfd7-4223-9559-221dd5541ff0","Handler":null,"Name":"","Endpoint":""} Jul 06 17:52:16.964198 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.964176 2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 17:52:16.964323 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:16.964223 2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 17:52:17.070188 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:17.070160 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:17.070324 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:17.070199 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:17.070324 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:17.070294 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:17.070471 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:17.070444 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:17.887367 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:17.887179 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:52:18.070703 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.070671 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:18.070871 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:18.070801 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:18.206883 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.206799 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"e6cc8fb8a9b04ffeb7a5910912c0f60316011562964b39433d337ff0fd38ee7d"} Jul 06 17:52:18.208765 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.208739 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" event={"ID":"798cac93-2f31-4157-b50e-b5281409046b","Type":"ContainerStarted","Data":"b53142ffe2fc9ca1b4d30329c4227d853398116c9397d8791c7e241cef80b660"} Jul 06 17:52:18.230361 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.230312 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cdt9b" podStartSLOduration=6.944118034 podStartE2EDuration="25.230294393s" totalImagesPullingTime="18.286176359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:16.217594689 +0000 UTC m=+23.747127040" watchObservedRunningTime="2026-07-06 17:52:18.230294393 +0000 UTC m=+25.759826754" Jul 06 17:52:18.230969 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.230935 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnh8r" podStartSLOduration=4.105104267 podStartE2EDuration="25.230919634s" totalImagesPullingTime="21.125815367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:18.230117527 +0000 UTC m=+25.759649887" watchObservedRunningTime="2026-07-06 17:52:18.230919634 +0000 UTC m=+25.760451995" Jul 06 17:52:18.452707 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.452675 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:52:18.453369 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:18.453354 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:52:19.070990 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:19.070958 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:19.071530 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:19.070958 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:19.071530 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:19.071095 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:19.071530 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:19.071129 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:19.211784 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:19.211760 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4crmj" Jul 06 17:52:19.274609 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:19.274569 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:20.070426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:20.070393 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:20.070614 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:20.070525 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:21.071051 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.070868 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:21.071748 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.070934 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:21.071748 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:21.071374 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:21.071748 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:21.071549 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:21.218543 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.218505 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" event={"ID":"7e14a4ba-0d02-4f44-a101-9f92ade15220","Type":"ContainerStarted","Data":"8826d3606ab04f24a47d99c0d4f7f4ac0c8a1f9aa8801fb3211f871e6d7446ce"} Jul 06 17:52:21.218872 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.218849 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:21.220451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.220431 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="1cc493d45e68ccdf1c0e3712b5543cd4281fdd7e2923f8fa8c0cfa6f600652da" exitCode=0 Jul 06 17:52:21.220561 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.220464 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"1cc493d45e68ccdf1c0e3712b5543cd4281fdd7e2923f8fa8c0cfa6f600652da"} Jul 06 17:52:21.234476 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.234455 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:21.255164 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:21.255119 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" podStartSLOduration=9.895250033 podStartE2EDuration="28.255102776s" totalImagesPullingTime="18.359852743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:21.254130617 +0000 UTC m=+28.783662979" watchObservedRunningTime="2026-07-06 17:52:21.255102776 +0000 UTC m=+28.784635113" Jul 06 17:52:22.070407 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.070380 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:22.070614 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:22.070480 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:22.222767 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.222707 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:22.223325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.222819 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:22.238097 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.238066 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:22.253875 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.253837 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:22.351956 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.351930 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:22.758672 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.758600 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ms7tt"] Jul 06 17:52:22.758818 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.758752 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:22.758859 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:22.758835 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:22.765594 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.765546 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s85lm"] Jul 06 17:52:22.765745 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.765667 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:22.765823 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:22.765779 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:22.766152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.766134 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c955d"] Jul 06 17:52:22.766315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:22.766231 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:22.766397 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:22.766319 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:23.226293 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:23.226257 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="70c9790eb145ebe68f35d8a07d48700cdcbeaf063bb151aa31da355fda51c651" exitCode=0 Jul 06 17:52:23.226750 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:23.226343 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"70c9790eb145ebe68f35d8a07d48700cdcbeaf063bb151aa31da355fda51c651"} Jul 06 17:52:23.375367 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:23.375178 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:23.438621 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:23.438585 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:24.070152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:24.070123 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:24.070152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:24.070138 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:24.070391 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:24.070253 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:24.070391 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:24.070268 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:24.070391 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:24.070330 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:24.070539 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:24.070399 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:24.230342 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:24.230309 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="6bff5d396805897aeb766cd362dc9a597413e7e9697f8575a9228b9446c8dca1" exitCode=0 Jul 06 17:52:24.230869 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:24.230386 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"6bff5d396805897aeb766cd362dc9a597413e7e9697f8575a9228b9446c8dca1"} Jul 06 17:52:25.516313 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:25.516261 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:25.618730 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:25.618669 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:25.951010 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:25.950969 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:26.059180 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.059142 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:26.070105 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.070080 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:26.070105 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.070099 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:26.070285 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.070109 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:26.070285 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.070207 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:26.070285 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.070268 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:26.070431 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.070332 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:26.656056 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.656025 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:26.656550 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.656182 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:26.656550 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.656250 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:58.656231793 +0000 UTC m=+66.185764131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 17:52:26.756366 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.756333 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:26.756613 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.756505 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:26.756613 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.756611 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret podName:df8e680c-48eb-41b1-a753-ac4688d2ef96 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:58.756557797 +0000 UTC m=+66.286090145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret") pod "global-pull-secret-syncer-ms7tt" (UID: "df8e680c-48eb-41b1-a753-ac4688d2ef96") : object "kube-system"/"original-pull-secret" not registered Jul 06 17:52:26.857350 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:26.857316 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:26.857561 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.857472 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 17:52:26.857561 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.857491 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 17:52:26.857561 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.857502 2587 projected.go:196] Error preparing data for projected volume kube-api-access-btqfk for pod openshift-network-diagnostics/network-check-target-s85lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:26.857801 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:26.857571 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk podName:f54bf57d-b686-446a-a5e5-4071f8027829 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:58.857544692 +0000 UTC m=+66.387077032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-btqfk" (UniqueName: "kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk") pod "network-check-target-s85lm" (UID: "f54bf57d-b686-446a-a5e5-4071f8027829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 17:52:28.070514 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.070477 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:28.070995 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.070599 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:28.070995 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:28.070611 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s85lm" podUID="f54bf57d-b686-446a-a5e5-4071f8027829" Jul 06 17:52:28.070995 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.070633 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:28.070995 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:28.070747 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ms7tt" podUID="df8e680c-48eb-41b1-a753-ac4688d2ef96" Jul 06 17:52:28.070995 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:28.070845 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c955d" podUID="051b52de-5f8c-4f51-88fb-0b286ab70161" Jul 06 17:52:28.276106 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.276024 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-13.ec2.internal" event="NodeReady" Jul 06 17:52:28.276268 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.276171 2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 17:52:28.336219 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.336185 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x88hn"] Jul 06 17:52:28.338975 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.338948 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.344918 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.344888 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pm8pm\"" Jul 06 17:52:28.345349 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.345326 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 17:52:28.345967 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.345944 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 17:52:28.346916 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.346185 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6qd"] Jul 06 17:52:28.348559 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.348540 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.352482 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.352450 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 17:52:28.352597 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.352498 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 17:52:28.352597 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.352566 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 17:52:28.352830 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.352814 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q7vmz\"" Jul 06 17:52:28.359546 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.359524 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x88hn"] Jul 06 17:52:28.366890 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.366866 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6qd"] Jul 06 17:52:28.469201 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469164 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwgm\" (UniqueName: \"kubernetes.io/projected/19b71253-28c0-43c8-9e13-3f7de5bd5af8-kube-api-access-ftwgm\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.469201 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469203 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.469432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469233 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx89\" (UniqueName: \"kubernetes.io/projected/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-kube-api-access-gbx89\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.469432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469348 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-cert\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.469432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469387 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b71253-28c0-43c8-9e13-3f7de5bd5af8-config-volume\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.469432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.469413 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b71253-28c0-43c8-9e13-3f7de5bd5af8-tmp-dir\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.547879 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.547784 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:28.570233 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570213 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwgm\" (UniqueName: \"kubernetes.io/projected/19b71253-28c0-43c8-9e13-3f7de5bd5af8-kube-api-access-ftwgm\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.570304 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570245 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.570304 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570267 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx89\" (UniqueName: \"kubernetes.io/projected/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-kube-api-access-gbx89\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.570304 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570293 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-cert\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.570456 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:28.570366 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:28.570456 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570404 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b71253-28c0-43c8-9e13-3f7de5bd5af8-config-volume\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.570456 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:28.570424 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:29.070404962 +0000 UTC m=+36.599937300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:28.570456 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570451 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b71253-28c0-43c8-9e13-3f7de5bd5af8-tmp-dir\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.570874 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570853 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b71253-28c0-43c8-9e13-3f7de5bd5af8-tmp-dir\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.570954 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.570939 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b71253-28c0-43c8-9e13-3f7de5bd5af8-config-volume\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.574557 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.574533 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-cert\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.579850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.579815 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwgm\" (UniqueName: \"kubernetes.io/projected/19b71253-28c0-43c8-9e13-3f7de5bd5af8-kube-api-access-ftwgm\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:28.581658 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.581632 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx89\" (UniqueName: \"kubernetes.io/projected/0fa8514e-3bda-47f6-bed9-f94e0a3682ce-kube-api-access-gbx89\") pod \"ingress-canary-6j6qd\" (UID: \"0fa8514e-3bda-47f6-bed9-f94e0a3682ce\") " pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.647449 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.647425 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:28.659477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.659461 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j6qd" Jul 06 17:52:28.791677 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:28.791513 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6qd"] Jul 06 17:52:28.794581 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:28.794542 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa8514e_3bda_47f6_bed9_f94e0a3682ce.slice/crio-dbdf52d31da7c277e759d9e52d47e17cdfd110dff6b52e20893c4bca3b448add WatchSource:0}: Error finding container dbdf52d31da7c277e759d9e52d47e17cdfd110dff6b52e20893c4bca3b448add: Status 404 returned error can't find the container with id dbdf52d31da7c277e759d9e52d47e17cdfd110dff6b52e20893c4bca3b448add Jul 06 17:52:29.074956 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:29.074931 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:29.075394 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:29.075035 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:29.075394 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:29.075081 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:30.075068172 +0000 UTC m=+37.604600510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:29.245287 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:29.245258 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="2d362308024935b0a0a08fcc1ce70fcea221bde324a2997534ab2917d9c0aaac" exitCode=0 Jul 06 17:52:29.245487 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:29.245350 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"2d362308024935b0a0a08fcc1ce70fcea221bde324a2997534ab2917d9c0aaac"} Jul 06 17:52:29.246567 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:29.246448 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j6qd" event={"ID":"0fa8514e-3bda-47f6-bed9-f94e0a3682ce","Type":"ContainerStarted","Data":"dbdf52d31da7c277e759d9e52d47e17cdfd110dff6b52e20893c4bca3b448add"} Jul 06 17:52:30.070440 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.070405 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:30.070440 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.070442 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:30.070734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.070442 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074498 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074515 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074537 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074537 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-22ww9\"" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074568 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 17:52:30.074679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.074517 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvljr\"" Jul 06 17:52:30.083186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.083164 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:30.083349 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:30.083330 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:30.083436 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:30.083395 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:32.083378195 +0000 UTC m=+39.612910533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:30.251581 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.251552 2587 generic.go:356] "Generic (PLEG): container finished" podID="d0316b37-42d5-4ac9-86a7-9baecdd16214" containerID="231339cdfb5ac74acf23b9e6afa23ff838188e967f979521174a4e87accc21be" exitCode=0 Jul 06 17:52:30.251759 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:30.251623 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerDied","Data":"231339cdfb5ac74acf23b9e6afa23ff838188e967f979521174a4e87accc21be"} Jul 06 17:52:31.255589 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.255354 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j6qd" event={"ID":"0fa8514e-3bda-47f6-bed9-f94e0a3682ce","Type":"ContainerStarted","Data":"46de34484e393acf3d66dad2bc248a7e53f7837fcf2363d563a58e790679fc2a"} Jul 06 17:52:31.258924 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.258887 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-928jl" event={"ID":"d0316b37-42d5-4ac9-86a7-9baecdd16214","Type":"ContainerStarted","Data":"6a2bc1a738c14041b194b277b28261e219f61dd32ac95bf8a2090ddca2389ecc"} Jul 06 17:52:31.277245 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.277202 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6j6qd" podStartSLOduration=0.937668756 podStartE2EDuration="3.277186199s" totalImagesPullingTime="2.339517443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:31.276622776 +0000 UTC m=+38.806155136" watchObservedRunningTime="2026-07-06 17:52:31.277186199 +0000 UTC m=+38.806718587" Jul 06 17:52:31.305970 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.305916 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-928jl" podStartSLOduration=8.263135928 podStartE2EDuration="38.305900534s" totalImagesPullingTime="30.042764606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:31.305401508 +0000 UTC m=+38.834933870" watchObservedRunningTime="2026-07-06 17:52:31.305900534 +0000 UTC m=+38.835432894" Jul 06 17:52:31.436594 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.436502 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:31.537005 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.536966 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:31.937878 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:31.937846 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:32.038005 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:32.037975 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:32.100755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:32.100705 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:32.100897 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:32.100854 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:32.100939 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:32.100919 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:36.100903562 +0000 UTC m=+43.630435900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:34.889916 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:34.889874 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:34.957485 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:34.957448 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:36.125932 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:36.125899 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:36.126390 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:36.126066 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:36.126390 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:36.126143 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:44.12612294 +0000 UTC m=+51.655655278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:44.176294 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:44.176256 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:52:44.176895 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:44.176441 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:52:44.176895 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:44.176520 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:00.176498972 +0000 UTC m=+67.706031313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:52:46.605771 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.605736 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jj9mh"] Jul 06 17:52:46.607705 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.607691 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.611263 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.611242 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 17:52:46.611475 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.611461 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 17:52:46.611770 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.611753 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 17:52:46.612021 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.612007 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 17:52:46.612553 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.612541 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d5w9d\"" Jul 06 17:52:46.617242 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.617227 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 17:52:46.626839 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.626812 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jj9mh"] Jul 06 17:52:46.689816 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.689778 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.689996 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.689833 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2qv\" (UniqueName: \"kubernetes.io/projected/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-kube-api-access-hd2qv\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.689996 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.689912 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-tmp\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.689996 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.689943 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-snapshots\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.689996 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.689993 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.690138 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.690010 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-serving-cert\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.731248 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.731210 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:46.790257 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790224 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-tmp\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790257 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790261 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-snapshots\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790307 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790331 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-serving-cert\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790365 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790578 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790535 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2qv\" (UniqueName: \"kubernetes.io/projected/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-kube-api-access-hd2qv\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.790995 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.790973 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-tmp\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.791093 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.791067 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-snapshots\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.791221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.791205 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.791725 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.791693 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.794437 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.794421 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-serving-cert\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.799357 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.799327 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2qv\" (UniqueName: \"kubernetes.io/projected/01cb4dd5-3f35-4e37-8b28-a849b62b9e70-kube-api-access-hd2qv\") pod \"insights-operator-5c847dcfc4-jj9mh\" (UID: \"01cb4dd5-3f35-4e37-8b28-a849b62b9e70\") " pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:46.830187 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.830159 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:46.916585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:46.916510 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" Jul 06 17:52:47.044440 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:47.044409 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jj9mh"] Jul 06 17:52:47.047102 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:47.047070 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cb4dd5_3f35_4e37_8b28_a849b62b9e70.slice/crio-43e6f3c743237768abd910ff5fbd27db62956dbe3d2e5b7dc2a36293d49918f6 WatchSource:0}: Error finding container 43e6f3c743237768abd910ff5fbd27db62956dbe3d2e5b7dc2a36293d49918f6: Status 404 returned error can't find the container with id 43e6f3c743237768abd910ff5fbd27db62956dbe3d2e5b7dc2a36293d49918f6 Jul 06 17:52:47.295387 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:47.295300 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" event={"ID":"01cb4dd5-3f35-4e37-8b28-a849b62b9e70","Type":"ContainerStarted","Data":"43e6f3c743237768abd910ff5fbd27db62956dbe3d2e5b7dc2a36293d49918f6"} Jul 06 17:52:48.505041 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.505010 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58"] Jul 06 17:52:48.507344 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.507319 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:48.510727 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.510683 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 17:52:48.511141 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.511119 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 17:52:48.511690 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.511345 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jkn8r\"" Jul 06 17:52:48.511690 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.511679 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:52:48.518647 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.518623 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58"] Jul 06 17:52:48.602881 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.602841 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:48.602881 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.602880 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknll\" (UniqueName: \"kubernetes.io/projected/bc6bac78-03a2-4366-8df3-a331874e9d94-kube-api-access-nknll\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:48.703532 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.703501 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknll\" (UniqueName: \"kubernetes.io/projected/bc6bac78-03a2-4366-8df3-a331874e9d94-kube-api-access-nknll\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:48.703669 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.703585 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:48.703709 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:48.703674 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:52:48.703764 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:48.703741 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:49.203708605 +0000 UTC m=+56.733240943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:52:48.712954 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:48.712934 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknll\" (UniqueName: \"kubernetes.io/projected/bc6bac78-03a2-4366-8df3-a331874e9d94-kube-api-access-nknll\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:49.207547 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.207500 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:49.207762 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:49.207631 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:52:49.207762 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:49.207728 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:50.207692885 +0000 UTC m=+57.737225227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:52:49.578782 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.578747 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz"] Jul 06 17:52:49.580522 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.580507 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" Jul 06 17:52:49.584986 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.584954 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 17:52:49.585949 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.585926 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qv72h\"" Jul 06 17:52:49.585949 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.585929 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:52:49.605989 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.605963 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz"] Jul 06 17:52:49.610432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.610408 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghv24\" (UniqueName: \"kubernetes.io/projected/13eea92f-ac58-4933-909b-3fcfcaa50252-kube-api-access-ghv24\") pod \"volume-data-source-validator-5d456fdbcd-td6wz\" (UID: \"13eea92f-ac58-4933-909b-3fcfcaa50252\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" Jul 06 17:52:49.711010 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.710936 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghv24\" (UniqueName: \"kubernetes.io/projected/13eea92f-ac58-4933-909b-3fcfcaa50252-kube-api-access-ghv24\") pod \"volume-data-source-validator-5d456fdbcd-td6wz\" (UID: \"13eea92f-ac58-4933-909b-3fcfcaa50252\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" Jul 06 17:52:49.719917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.719885 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghv24\" (UniqueName: \"kubernetes.io/projected/13eea92f-ac58-4933-909b-3fcfcaa50252-kube-api-access-ghv24\") pod \"volume-data-source-validator-5d456fdbcd-td6wz\" (UID: \"13eea92f-ac58-4933-909b-3fcfcaa50252\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" Jul 06 17:52:49.889274 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:49.889227 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" Jul 06 17:52:50.010406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.010377 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz"] Jul 06 17:52:50.013762 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:50.013735 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eea92f_ac58_4933_909b_3fcfcaa50252.slice/crio-d237048b4a11bb960812bbff60bfb32d1678cd96c2cec411c3dad177d1b3cceb WatchSource:0}: Error finding container d237048b4a11bb960812bbff60bfb32d1678cd96c2cec411c3dad177d1b3cceb: Status 404 returned error can't find the container with id d237048b4a11bb960812bbff60bfb32d1678cd96c2cec411c3dad177d1b3cceb Jul 06 17:52:50.215181 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.215097 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:50.215329 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.215242 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:52:50.215329 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.215305 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:52.215290705 +0000 UTC m=+59.744823043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:52:50.302776 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.302745 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" event={"ID":"13eea92f-ac58-4933-909b-3fcfcaa50252","Type":"ContainerStarted","Data":"d237048b4a11bb960812bbff60bfb32d1678cd96c2cec411c3dad177d1b3cceb"} Jul 06 17:52:50.304111 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.304086 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" event={"ID":"01cb4dd5-3f35-4e37-8b28-a849b62b9e70","Type":"ContainerStarted","Data":"f727fd0f7a9436fb29b49178b8fe3f52ffbff6ad94b273ed40b46fa23e26a755"} Jul 06 17:52:50.321854 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.321801 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" podStartSLOduration=1.9558083609999999 podStartE2EDuration="4.321783214s" totalImagesPullingTime="2.365974853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:50.321646038 +0000 UTC m=+57.851178398" watchObservedRunningTime="2026-07-06 17:52:50.321783214 +0000 UTC m=+57.851315576" Jul 06 17:52:50.392945 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.392914 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:50.490232 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.490141 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:50.511196 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.511164 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lc5tx"] Jul 06 17:52:50.513541 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.513523 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.520702 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.520685 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 17:52:50.521237 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.521001 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:52:50.521237 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.521028 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 17:52:50.521237 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.521131 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 17:52:50.521237 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.521146 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rgsmj\"" Jul 06 17:52:50.526390 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.526372 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 17:52:50.528665 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.528635 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lc5tx"] Jul 06 17:52:50.618565 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.618528 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d22ec3d0-b287-4110-a439-f96f099270d3-serving-cert\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.618937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.618602 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-trusted-ca\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.618937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.618629 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7tj\" (UniqueName: \"kubernetes.io/projected/d22ec3d0-b287-4110-a439-f96f099270d3-kube-api-access-5g7tj\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.618937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.618667 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-config\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.627208 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.627184 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zcxfd"] Jul 06 17:52:50.629208 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.629189 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.634042 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.634024 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 17:52:50.634274 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.634251 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j6zr4\"" Jul 06 17:52:50.634405 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.634344 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 17:52:50.651349 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.651325 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zcxfd"] Jul 06 17:52:50.678600 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.678578 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:50.719106 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719079 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-trusted-ca\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.719218 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719110 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7tj\" (UniqueName: \"kubernetes.io/projected/d22ec3d0-b287-4110-a439-f96f099270d3-kube-api-access-5g7tj\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.719218 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719133 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-data-volume\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.719300 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719260 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4wt\" (UniqueName: \"kubernetes.io/projected/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-api-access-7n4wt\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.719343 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719304 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-config\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.719343 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719330 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.719422 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719406 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-crio-socket\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.719457 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719442 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.719547 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719529 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d22ec3d0-b287-4110-a439-f96f099270d3-serving-cert\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.719962 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719943 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-trusted-ca\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.720037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.719972 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d22ec3d0-b287-4110-a439-f96f099270d3-config\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.721738 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.721702 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d22ec3d0-b287-4110-a439-f96f099270d3-serving-cert\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.730705 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.730684 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7tj\" (UniqueName: \"kubernetes.io/projected/d22ec3d0-b287-4110-a439-f96f099270d3-kube-api-access-5g7tj\") pod \"console-operator-68b757fb67-lc5tx\" (UID: \"d22ec3d0-b287-4110-a439-f96f099270d3\") " pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.820248 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820215 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-crio-socket\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820257 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820328 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-data-volume\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820355 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4wt\" (UniqueName: \"kubernetes.io/projected/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-api-access-7n4wt\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820394 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820425 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.820407 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 17:52:50.820635 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.820485 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:51.320464492 +0000 UTC m=+58.849996830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : configmap "kube-rbac-proxy" not found Jul 06 17:52:50.820635 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.820538 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:52:50.820635 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:50.820587 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:51.320572029 +0000 UTC m=+58.850104380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:52:50.820635 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820406 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-crio-socket\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.820827 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.820656 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-data-volume\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.822131 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.822109 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:52:50.840118 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.840093 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4wt\" (UniqueName: \"kubernetes.io/projected/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-api-access-7n4wt\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:50.947795 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:50.947766 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lc5tx"] Jul 06 17:52:50.951276 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:50.951237 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ec3d0_b287_4110_a439_f96f099270d3.slice/crio-2071888a681b80f89bd76666a8142364647a7359b1b1b2f437eee7154d3d2e9a WatchSource:0}: Error finding container 2071888a681b80f89bd76666a8142364647a7359b1b1b2f437eee7154d3d2e9a: Status 404 returned error can't find the container with id 2071888a681b80f89bd76666a8142364647a7359b1b1b2f437eee7154d3d2e9a Jul 06 17:52:51.308488 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:51.308436 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" event={"ID":"d22ec3d0-b287-4110-a439-f96f099270d3","Type":"ContainerStarted","Data":"2071888a681b80f89bd76666a8142364647a7359b1b1b2f437eee7154d3d2e9a"} Jul 06 17:52:51.325262 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:51.325233 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:51.325447 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:51.325348 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:51.325447 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:51.325400 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:52:51.325447 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:51.325426 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 17:52:51.325632 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:51.325471 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:52.325450682 +0000 UTC m=+59.854983021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:52:51.325632 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:51.325490 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:52.325481065 +0000 UTC m=+59.855013408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : configmap "kube-rbac-proxy" not found Jul 06 17:52:52.233543 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.233499 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:52.233993 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.233691 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:52:52.233993 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.233785 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:56.233763303 +0000 UTC m=+63.763295651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:52:52.312059 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.312030 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" event={"ID":"13eea92f-ac58-4933-909b-3fcfcaa50252","Type":"ContainerStarted","Data":"9c9887f232e8bb8e0a03d8920158c6c7b9998cea4d97bf30c71bba1a9940b1d7"} Jul 06 17:52:52.328492 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.328443 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-td6wz" podStartSLOduration=1.969422542 podStartE2EDuration="3.32842858s" totalImagesPullingTime="1.359006038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:52:52.328083768 +0000 UTC m=+59.857616152" watchObservedRunningTime="2026-07-06 17:52:52.32842858 +0000 UTC m=+59.857960940" Jul 06 17:52:52.334354 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.334335 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:52.334453 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.334401 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:52.334491 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.334460 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 17:52:52.334524 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.334512 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:52:52.334524 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.334519 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:54.334504081 +0000 UTC m=+61.864036435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : configmap "kube-rbac-proxy" not found Jul 06 17:52:52.334592 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:52.334546 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:54.334535688 +0000 UTC m=+61.864068028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:52:52.413961 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.413928 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:52.430250 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.430226 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lnnp_59cb332e-0f1b-4204-9c67-4df28a5ddbc3/dns-node-resolver/0.log" Jul 06 17:52:52.477504 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.477474 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:52.954994 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.954961 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:52:52.982679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.982646 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:52:52.982861 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.982776 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:52.985815 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.985543 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 17:52:52.985815 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.985574 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 17:52:52.985815 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.985554 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-74rbm\"" Jul 06 17:52:52.985815 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.985697 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 17:52:52.991047 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:52.990769 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 17:52:53.039451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039427 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039471 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039519 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdczd\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039553 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039577 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039659 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039696 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.039755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.039726 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.140692 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.140654 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.140871 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.140735 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdczd\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.140871 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.140778 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.140871 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.140851 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141028 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.140947 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141078 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.141027 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:52:53.141078 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.141041 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:52:53.141176 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141084 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141176 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.141098 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:52:53.641080521 +0000 UTC m=+61.170612861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:52:53.141176 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141141 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141176 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141167 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141401 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141381 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141462 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141447 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.141971 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.141948 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.143418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.143398 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.143418 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.143407 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.153856 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.153835 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.153929 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.153863 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdczd\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.626330 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.626308 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfbkd_efb406b4-8fd5-44d2-a657-6f6691c098a2/node-ca/0.log" Jul 06 17:52:53.644942 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:53.644917 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:53.645070 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.645054 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:52:53.645111 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.645072 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:52:53.645144 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:53.645120 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:52:54.645104609 +0000 UTC m=+62.174636947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:52:54.227758 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.227728 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6j6qd_0fa8514e-3bda-47f6-bed9-f94e0a3682ce/serve-healthcheck-canary/0.log" Jul 06 17:52:54.240314 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.240292 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hb6qz" Jul 06 17:52:54.318523 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.318494 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/0.log" Jul 06 17:52:54.318684 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.318535 2587 generic.go:356] "Generic (PLEG): container finished" podID="d22ec3d0-b287-4110-a439-f96f099270d3" containerID="0a02d47e52d8c5d3c4b02adf99ad086b285b3721e66abee585a6d2ac17dd6caa" exitCode=255 Jul 06 17:52:54.318684 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.318590 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" event={"ID":"d22ec3d0-b287-4110-a439-f96f099270d3","Type":"ContainerDied","Data":"0a02d47e52d8c5d3c4b02adf99ad086b285b3721e66abee585a6d2ac17dd6caa"} Jul 06 17:52:54.318811 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.318799 2587 scope.go:117] "RemoveContainer" containerID="0a02d47e52d8c5d3c4b02adf99ad086b285b3721e66abee585a6d2ac17dd6caa" Jul 06 17:52:54.350557 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.350529 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:54.350664 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.350616 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 17:52:54.351011 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.350993 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:58.350970065 +0000 UTC m=+65.880502405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : configmap "kube-rbac-proxy" not found Jul 06 17:52:54.352977 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.351615 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:54.352977 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.352343 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:52:54.352977 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.352397 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:52:58.35238156 +0000 UTC m=+65.881913898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:52:54.380893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.380870 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:54.486580 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.486546 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:54.654238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:54.654203 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:54.654601 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.654361 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:52:54.654601 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.654378 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:52:54.654601 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:54.654447 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:52:56.654431113 +0000 UTC m=+64.183963453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:52:55.322819 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.322794 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/1.log" Jul 06 17:52:55.323217 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.323201 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/0.log" Jul 06 17:52:55.323275 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.323237 2587 generic.go:356] "Generic (PLEG): container finished" podID="d22ec3d0-b287-4110-a439-f96f099270d3" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" exitCode=255 Jul 06 17:52:55.323310 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.323267 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" event={"ID":"d22ec3d0-b287-4110-a439-f96f099270d3","Type":"ContainerDied","Data":"fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea"} Jul 06 17:52:55.323343 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.323323 2587 scope.go:117] "RemoveContainer" containerID="0a02d47e52d8c5d3c4b02adf99ad086b285b3721e66abee585a6d2ac17dd6caa" Jul 06 17:52:55.323517 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:55.323499 2587 scope.go:117] "RemoveContainer" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" Jul 06 17:52:55.323742 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:55.323701 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lc5tx_openshift-console-operator(d22ec3d0-b287-4110-a439-f96f099270d3)\"" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" Jul 06 17:52:56.265887 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:56.265831 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:52:56.266309 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.265987 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:52:56.266309 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.266049 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:04.266033008 +0000 UTC m=+71.795565347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:52:56.327375 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:56.327351 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/1.log" Jul 06 17:52:56.327670 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:56.327646 2587 scope.go:117] "RemoveContainer" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" Jul 06 17:52:56.327838 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.327819 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lc5tx_openshift-console-operator(d22ec3d0-b287-4110-a439-f96f099270d3)\"" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" Jul 06 17:52:56.669351 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:56.669316 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:52:56.669521 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.669449 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:52:56.669521 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.669467 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:52:56.669521 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:56.669519 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:53:00.669505387 +0000 UTC m=+68.199037726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:52:58.386067 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.386033 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:58.386450 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.386130 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:52:58.386450 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.386183 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:52:58.386450 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.386197 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 17:52:58.386450 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.386255 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:06.38623586 +0000 UTC m=+73.915768199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:52:58.386450 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.386276 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:06.386266667 +0000 UTC m=+73.915799007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : configmap "kube-rbac-proxy" not found Jul 06 17:52:58.689016 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.688929 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:52:58.691688 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.691672 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 17:52:58.699692 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.699676 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 17:52:58.699804 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:52:58.699738 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs podName:051b52de-5f8c-4f51-88fb-0b286ab70161 nodeName:}" failed. No retries permitted until 2026-07-06 17:54:02.699725006 +0000 UTC m=+130.229257357 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs") pod "network-metrics-daemon-c955d" (UID: "051b52de-5f8c-4f51-88fb-0b286ab70161") : secret "metrics-daemon-secret" not found Jul 06 17:52:58.789830 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.789795 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:58.792454 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.792438 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 17:52:58.802666 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.802645 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df8e680c-48eb-41b1-a753-ac4688d2ef96-original-pull-secret\") pod \"global-pull-secret-syncer-ms7tt\" (UID: \"df8e680c-48eb-41b1-a753-ac4688d2ef96\") " pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:58.883219 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.883193 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ms7tt" Jul 06 17:52:58.890750 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.890702 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:58.893851 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.893829 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 17:52:58.903517 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.903496 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 17:52:58.914017 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:58.913993 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/f54bf57d-b686-446a-a5e5-4071f8027829-kube-api-access-btqfk\") pod \"network-check-target-s85lm\" (UID: \"f54bf57d-b686-446a-a5e5-4071f8027829\") " pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:59.004086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.004054 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ms7tt"] Jul 06 17:52:59.006707 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:59.006681 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8e680c_48eb_41b1_a753_ac4688d2ef96.slice/crio-0aee79f9ccbadfad8a53dbe3e77b0319d19b20959bea6e09637190461414fde9 WatchSource:0}: Error finding container 0aee79f9ccbadfad8a53dbe3e77b0319d19b20959bea6e09637190461414fde9: Status 404 returned error can't find the container with id 0aee79f9ccbadfad8a53dbe3e77b0319d19b20959bea6e09637190461414fde9 Jul 06 17:52:59.193293 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.193264 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvljr\"" Jul 06 17:52:59.201434 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.201410 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:52:59.334431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.334404 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s85lm"] Jul 06 17:52:59.336633 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.336608 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ms7tt" event={"ID":"df8e680c-48eb-41b1-a753-ac4688d2ef96","Type":"ContainerStarted","Data":"0aee79f9ccbadfad8a53dbe3e77b0319d19b20959bea6e09637190461414fde9"} Jul 06 17:52:59.337329 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:52:59.337308 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54bf57d_b686_446a_a5e5_4071f8027829.slice/crio-8976c8b7a5ac50fdf5c11c2932740d997243f7d9566cd94c51f07b253b8d3837 WatchSource:0}: Error finding container 8976c8b7a5ac50fdf5c11c2932740d997243f7d9566cd94c51f07b253b8d3837: Status 404 returned error can't find the container with id 8976c8b7a5ac50fdf5c11c2932740d997243f7d9566cd94c51f07b253b8d3837 Jul 06 17:52:59.417771 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.417739 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:59.529333 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.529242 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:52:59.618625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:52:59.618597 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:00.203344 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.203299 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:00.203550 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.203516 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 17:53:00.203632 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.203597 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls podName:19b71253-28c0-43c8-9e13-3f7de5bd5af8 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:32.203575999 +0000 UTC m=+99.733108343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls") pod "dns-default-x88hn" (UID: "19b71253-28c0-43c8-9e13-3f7de5bd5af8") : secret "dns-default-metrics-tls" not found Jul 06 17:53:00.341677 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.341641 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s85lm" event={"ID":"f54bf57d-b686-446a-a5e5-4071f8027829","Type":"ContainerStarted","Data":"8976c8b7a5ac50fdf5c11c2932740d997243f7d9566cd94c51f07b253b8d3837"} Jul 06 17:53:00.389067 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.389033 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bplpn"] Jul 06 17:53:00.409846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.409475 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bplpn"] Jul 06 17:53:00.409846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.409620 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.412796 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.412568 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sxg2f\"" Jul 06 17:53:00.412796 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.412678 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 17:53:00.412962 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.412829 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 17:53:00.506149 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.506065 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.506518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.506192 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8331cd19-bd71-41d3-97df-bf8a2bf42348-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.606692 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.606655 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.606887 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.606769 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8331cd19-bd71-41d3-97df-bf8a2bf42348-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.606887 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.606833 2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 17:53:00.606999 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.606901 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert podName:8331cd19-bd71-41d3-97df-bf8a2bf42348 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:01.106882602 +0000 UTC m=+68.636414947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bplpn" (UID: "8331cd19-bd71-41d3-97df-bf8a2bf42348") : secret "networking-console-plugin-cert" not found Jul 06 17:53:00.607458 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.607436 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8331cd19-bd71-41d3-97df-bf8a2bf42348-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:00.707992 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.707943 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:00.708258 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.708235 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:53:00.708258 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.708252 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:53:00.708447 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.708315 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:53:08.708294965 +0000 UTC m=+76.237827302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:53:00.823093 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.823062 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:00.823288 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.823129 2587 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:00.823504 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:00.823487 2587 scope.go:117] "RemoveContainer" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" Jul 06 17:53:00.823695 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:00.823675 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lc5tx_openshift-console-operator(d22ec3d0-b287-4110-a439-f96f099270d3)\"" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" Jul 06 17:53:01.112550 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:01.112455 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:01.112754 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:01.112630 2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 17:53:01.112754 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:01.112735 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert podName:8331cd19-bd71-41d3-97df-bf8a2bf42348 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:02.11269591 +0000 UTC m=+69.642228261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bplpn" (UID: "8331cd19-bd71-41d3-97df-bf8a2bf42348") : secret "networking-console-plugin-cert" not found Jul 06 17:53:02.122628 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:02.122597 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:02.123115 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:02.122774 2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 17:53:02.123115 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:02.122854 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert podName:8331cd19-bd71-41d3-97df-bf8a2bf42348 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:04.122835556 +0000 UTC m=+71.652367913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bplpn" (UID: "8331cd19-bd71-41d3-97df-bf8a2bf42348") : secret "networking-console-plugin-cert" not found Jul 06 17:53:02.451865 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:02.451774 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:02.556306 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:02.556272 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:02.948952 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:02.948914 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.020487 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.020439 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.185364 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.185319 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.282314 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.282219 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.647461 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.647419 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.751094 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.751060 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:03.884931 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:03.884897 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:04.142333 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.142225 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:04.142416 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:04.142351 2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 17:53:04.142416 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:04.142400 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert podName:8331cd19-bd71-41d3-97df-bf8a2bf42348 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:08.14238571 +0000 UTC m=+75.671918047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bplpn" (UID: "8331cd19-bd71-41d3-97df-bf8a2bf42348") : secret "networking-console-plugin-cert" not found Jul 06 17:53:04.343624 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.343537 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:53:04.344001 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:04.343658 2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 17:53:04.344001 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:04.343731 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls podName:bc6bac78-03a2-4366-8df3-a331874e9d94 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:20.343700889 +0000 UTC m=+87.873233229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2rh58" (UID: "bc6bac78-03a2-4366-8df3-a331874e9d94") : secret "samples-operator-tls" not found Jul 06 17:53:04.355126 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.355100 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ms7tt" event={"ID":"df8e680c-48eb-41b1-a753-ac4688d2ef96","Type":"ContainerStarted","Data":"af96100c9dac2c9f9bf027956e902378d8d45111002ea870e353463fcfe14437"} Jul 06 17:53:04.356515 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.356495 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s85lm" event={"ID":"f54bf57d-b686-446a-a5e5-4071f8027829","Type":"ContainerStarted","Data":"862ca196864f9e221ce0e9106dad738fd43e63f7b6491b95f285089e64632d9b"} Jul 06 17:53:04.356628 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.356617 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:53:04.371332 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.371294 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ms7tt" podStartSLOduration=66.292234258 podStartE2EDuration="1m11.37128111s" totalImagesPullingTime="5.079046852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:04.370622233 +0000 UTC m=+71.900154612" watchObservedRunningTime="2026-07-06 17:53:04.37128111 +0000 UTC m=+71.900813470" Jul 06 17:53:04.387112 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:04.387076 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s85lm" podStartSLOduration=66.64137203 podStartE2EDuration="1m11.387062742s" totalImagesPullingTime="4.745690712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:04.3866826 +0000 UTC m=+71.916214963" watchObservedRunningTime="2026-07-06 17:53:04.387062742 +0000 UTC m=+71.916595123" Jul 06 17:53:05.286610 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:05.286573 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:05.353103 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:05.353055 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:06.352140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.352089 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:06.460860 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.460832 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:06.461238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.460930 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:06.461238 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:06.460977 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 17:53:06.461238 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:06.461033 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls podName:dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:22.461020703 +0000 UTC m=+89.990553042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zcxfd" (UID: "dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5") : secret "insights-runtime-extractor-tls" not found Jul 06 17:53:06.461401 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.461384 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:06.776181 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.776098 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:06.962112 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:06.962071 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:07.074490 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:07.074451 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:07.199957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:07.199926 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:07.321357 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:07.321312 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:07.516241 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:07.516154 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:08.175137 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:08.175099 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:08.175357 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:08.175273 2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 17:53:08.175429 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:08.175366 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert podName:8331cd19-bd71-41d3-97df-bf8a2bf42348 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:16.175342372 +0000 UTC m=+83.704874730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bplpn" (UID: "8331cd19-bd71-41d3-97df-bf8a2bf42348") : secret "networking-console-plugin-cert" not found Jul 06 17:53:08.302279 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:08.302234 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:08.471192 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:08.471082 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:08.781534 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:08.781455 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:08.781913 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:08.781602 2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 17:53:08.781913 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:08.781619 2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-868bb7f4b8-7gmdw: secret "image-registry-tls" not found Jul 06 17:53:08.781913 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:08.781674 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls podName:3cae64d9-0470-4dec-98d3-ec42f366fefb nodeName:}" failed. No retries permitted until 2026-07-06 17:53:24.781658268 +0000 UTC m=+92.311190605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls") pod "image-registry-868bb7f4b8-7gmdw" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb") : secret "image-registry-tls" not found Jul 06 17:53:09.055265 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:09.055234 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:09.262175 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:09.262143 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:09.864764 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:09.864727 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:10.269188 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:10.269082 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:10.402776 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:10.402739 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:10.502565 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:10.502532 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:10.601275 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:10.601246 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:10.701564 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:10.701532 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:11.077789 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:11.077762 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:11.270344 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:11.270314 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:11.713743 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:11.713698 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:11.820146 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:11.820115 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:11.906069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:11.906037 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:12.070642 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.070620 2587 scope.go:117] "RemoveContainer" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" Jul 06 17:53:12.387033 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.386960 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 17:53:12.387406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.387334 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/1.log" Jul 06 17:53:12.387406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.387362 2587 generic.go:356] "Generic (PLEG): container finished" podID="d22ec3d0-b287-4110-a439-f96f099270d3" containerID="bb7f2465bc4c64eac84491cba00445955ca090d74ed7559a3232067fae9b1c57" exitCode=255 Jul 06 17:53:12.387501 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.387423 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" event={"ID":"d22ec3d0-b287-4110-a439-f96f099270d3","Type":"ContainerDied","Data":"bb7f2465bc4c64eac84491cba00445955ca090d74ed7559a3232067fae9b1c57"} Jul 06 17:53:12.387501 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.387470 2587 scope.go:117] "RemoveContainer" containerID="fa2234eca19dfcc7a7237aea0b8de456f01fec87779f36f9ac0577e53e2197ea" Jul 06 17:53:12.387805 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.387789 2587 scope.go:117] "RemoveContainer" containerID="bb7f2465bc4c64eac84491cba00445955ca090d74ed7559a3232067fae9b1c57" Jul 06 17:53:12.387986 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:12.387966 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-lc5tx_openshift-console-operator(d22ec3d0-b287-4110-a439-f96f099270d3)\"" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" Jul 06 17:53:12.504201 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.504165 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:12.666274 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.666192 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:12.882240 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:12.882205 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:13.057098 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:13.057067 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:13.134225 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:13.134186 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:13.392073 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:13.391996 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 17:53:14.071518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:14.071483 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:14.460955 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:14.460809 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:14.665583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:14.665537 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:14.867708 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:14.867677 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:15.265246 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:15.265154 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:15.875665 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:15.875632 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:16.244286 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.244199 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:16.246598 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.246568 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8331cd19-bd71-41d3-97df-bf8a2bf42348-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bplpn\" (UID: \"8331cd19-bd71-41d3-97df-bf8a2bf42348\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:16.266403 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.266371 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:16.320518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.320489 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" Jul 06 17:53:16.443204 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.443151 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bplpn"] Jul 06 17:53:16.445569 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:16.445540 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8331cd19_bd71_41d3_97df_bf8a2bf42348.slice/crio-0a4a66c5d5c84830f96c3e01501b83dd1693805a1e33c39162711480c1d22c83 WatchSource:0}: Error finding container 0a4a66c5d5c84830f96c3e01501b83dd1693805a1e33c39162711480c1d22c83: Status 404 returned error can't find the container with id 0a4a66c5d5c84830f96c3e01501b83dd1693805a1e33c39162711480c1d22c83 Jul 06 17:53:16.463860 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:16.463838 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:17.088233 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:17.088195 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:17.336843 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:17.336787 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:17.407018 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:17.406933 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" event={"ID":"8331cd19-bd71-41d3-97df-bf8a2bf42348","Type":"ContainerStarted","Data":"0a4a66c5d5c84830f96c3e01501b83dd1693805a1e33c39162711480c1d22c83"} Jul 06 17:53:17.519677 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:17.519641 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:18.332748 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:18.332703 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:18.412662 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:18.412631 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" event={"ID":"8331cd19-bd71-41d3-97df-bf8a2bf42348","Type":"ContainerStarted","Data":"0921d611c81e73257dff57d967b0b41308726f97158095e2c306aff7933719d4"} Jul 06 17:53:18.446842 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:18.446791 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bplpn" podStartSLOduration=17.366495928 podStartE2EDuration="18.446775145s" totalImagesPullingTime="1.080279217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:18.44599532 +0000 UTC m=+85.975527679" watchObservedRunningTime="2026-07-06 17:53:18.446775145 +0000 UTC m=+85.976307505" Jul 06 17:53:20.379876 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.379845 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:53:20.382355 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.382329 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc6bac78-03a2-4366-8df3-a331874e9d94-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2rh58\" (UID: \"bc6bac78-03a2-4366-8df3-a331874e9d94\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:53:20.622096 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.622063 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jkn8r\"" Jul 06 17:53:20.629694 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.629673 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" Jul 06 17:53:20.753198 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.753041 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58"] Jul 06 17:53:20.822405 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.822382 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:20.822492 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.822435 2587 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:20.822739 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:20.822706 2587 scope.go:117] "RemoveContainer" containerID="bb7f2465bc4c64eac84491cba00445955ca090d74ed7559a3232067fae9b1c57" Jul 06 17:53:20.822879 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:20.822862 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-lc5tx_openshift-console-operator(d22ec3d0-b287-4110-a439-f96f099270d3)\"" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" Jul 06 17:53:21.334244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:21.334214 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:21.422217 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:21.422186 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" event={"ID":"bc6bac78-03a2-4366-8df3-a331874e9d94","Type":"ContainerStarted","Data":"eec484c5e2079045fcf359490bb5dc5a8174c39e532102dee383cd9a1d45ae2b"} Jul 06 17:53:21.434779 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:21.434745 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:21.529991 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:21.529958 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:21.636167 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:21.636070 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:22.493465 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:22.493431 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:22.495756 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:22.495733 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcxfd\" (UID: \"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5\") " pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:22.745407 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:22.745338 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j6zr4\"" Jul 06 17:53:22.750285 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:22.750265 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zcxfd" Jul 06 17:53:22.879900 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:22.879875 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zcxfd"] Jul 06 17:53:22.882375 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:22.882347 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc91d5d4_5bc5_4abf_abfe_4045ae0ce8a5.slice/crio-c10c81e11beb1d053c42edec198e399d2f8b246b9b9b73e46d0d23a6ada9efee WatchSource:0}: Error finding container c10c81e11beb1d053c42edec198e399d2f8b246b9b9b73e46d0d23a6ada9efee: Status 404 returned error can't find the container with id c10c81e11beb1d053c42edec198e399d2f8b246b9b9b73e46d0d23a6ada9efee Jul 06 17:53:23.431256 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:23.431219 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcxfd" event={"ID":"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5","Type":"ContainerStarted","Data":"c10c81e11beb1d053c42edec198e399d2f8b246b9b9b73e46d0d23a6ada9efee"} Jul 06 17:53:24.438838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:24.438796 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" event={"ID":"bc6bac78-03a2-4366-8df3-a331874e9d94","Type":"ContainerStarted","Data":"c20ce69fbf3704061578ced52189ca87d27d4a8fb4e971fca1c587737348299f"} Jul 06 17:53:24.439405 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:24.438869 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" event={"ID":"bc6bac78-03a2-4366-8df3-a331874e9d94","Type":"ContainerStarted","Data":"e53685332e082d71d0b4b54d0d5aee96e6fd87c54b4d6e631950ecc89f9f9907"} Jul 06 17:53:24.480580 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:24.480476 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2rh58" podStartSLOduration=33.424326412 podStartE2EDuration="36.480453652s" totalImagesPullingTime="3.05612724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:24.478806424 +0000 UTC m=+92.008338809" watchObservedRunningTime="2026-07-06 17:53:24.480453652 +0000 UTC m=+92.009986013" Jul 06 17:53:24.815456 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:24.815415 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:24.817997 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:24.817970 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"image-registry-868bb7f4b8-7gmdw\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:25.094606 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:25.094521 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:26.693185 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:26.693135 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:26.771557 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:26.771518 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:27.104957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.104871 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:53:27.107687 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:27.107660 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cae64d9_0470_4dec_98d3_ec42f366fefb.slice/crio-66e4d38b0388bd83e049494556034142ec64c878b9b595eec68b42158ea1a368 WatchSource:0}: Error finding container 66e4d38b0388bd83e049494556034142ec64c878b9b595eec68b42158ea1a368: Status 404 returned error can't find the container with id 66e4d38b0388bd83e049494556034142ec64c878b9b595eec68b42158ea1a368 Jul 06 17:53:27.451199 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.451159 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcxfd" event={"ID":"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5","Type":"ContainerStarted","Data":"2bdc6769c676d552d29b3165ad00a6adb55e983fded24b37eeb182b846dba473"} Jul 06 17:53:27.452917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.452883 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" event={"ID":"3cae64d9-0470-4dec-98d3-ec42f366fefb","Type":"ContainerStarted","Data":"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027"} Jul 06 17:53:27.453058 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.452933 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" event={"ID":"3cae64d9-0470-4dec-98d3-ec42f366fefb","Type":"ContainerStarted","Data":"66e4d38b0388bd83e049494556034142ec64c878b9b595eec68b42158ea1a368"} Jul 06 17:53:27.453058 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.453023 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:27.484648 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:27.484596 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" podStartSLOduration=35.484578107 podStartE2EDuration="35.484578107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:27.483756657 +0000 UTC m=+95.013289011" watchObservedRunningTime="2026-07-06 17:53:27.484578107 +0000 UTC m=+95.014110465" Jul 06 17:53:28.458855 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:28.458809 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcxfd" event={"ID":"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5","Type":"ContainerStarted","Data":"74d3fc55cc150c586f2067ef9efa2aa3be3912ebd1abddb5a70ec35f6aa55540"} Jul 06 17:53:29.868319 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.868283 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm"] Jul 06 17:53:29.880900 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.880867 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:53:29.881069 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.880981 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:29.891797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.891692 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm"] Jul 06 17:53:29.895444 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.895420 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 17:53:29.895915 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.895735 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 17:53:29.896156 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.896132 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 17:53:29.896271 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:29.896251 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 17:53:30.025408 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.025360 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:30.059398 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.059355 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a88d641-d614-472c-a210-aba0a2e09fc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.059577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.059445 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a88d641-d614-472c-a210-aba0a2e09fc8-tmp\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.059577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.059487 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq928\" (UniqueName: \"kubernetes.io/projected/8a88d641-d614-472c-a210-aba0a2e09fc8-kube-api-access-lq928\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.160082 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.160008 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a88d641-d614-472c-a210-aba0a2e09fc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.160222 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.160103 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a88d641-d614-472c-a210-aba0a2e09fc8-tmp\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.160222 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.160139 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq928\" (UniqueName: \"kubernetes.io/projected/8a88d641-d614-472c-a210-aba0a2e09fc8-kube-api-access-lq928\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.160495 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.160467 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a88d641-d614-472c-a210-aba0a2e09fc8-tmp\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.162630 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.162606 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a88d641-d614-472c-a210-aba0a2e09fc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.172094 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.172072 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq928\" (UniqueName: \"kubernetes.io/projected/8a88d641-d614-472c-a210-aba0a2e09fc8-kube-api-access-lq928\") pod \"klusterlet-addon-workmgr-666564b8d6-h7rqm\" (UID: \"8a88d641-d614-472c-a210-aba0a2e09fc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.175391 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.175364 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:30.193653 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.193618 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:30.359311 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.359284 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm"] Jul 06 17:53:30.367117 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:30.367089 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a88d641_d614_472c_a210_aba0a2e09fc8.slice/crio-4ad317cbb1b74c039162bc271c452316f1acbaab2be1b16b41a980a25dc232fd WatchSource:0}: Error finding container 4ad317cbb1b74c039162bc271c452316f1acbaab2be1b16b41a980a25dc232fd: Status 404 returned error can't find the container with id 4ad317cbb1b74c039162bc271c452316f1acbaab2be1b16b41a980a25dc232fd Jul 06 17:53:30.465794 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.465687 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" event={"ID":"8a88d641-d614-472c-a210-aba0a2e09fc8","Type":"ContainerStarted","Data":"4ad317cbb1b74c039162bc271c452316f1acbaab2be1b16b41a980a25dc232fd"} Jul 06 17:53:30.467604 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.467581 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcxfd" event={"ID":"dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5","Type":"ContainerStarted","Data":"243c46d1fcc287950dfc831f18f9e093fa380f843bbbc02b831c091d90c91f48"} Jul 06 17:53:30.518804 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:30.518678 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zcxfd" podStartSLOduration=33.28987558 podStartE2EDuration="40.518661219s" totalImagesPullingTime="7.228785639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:52:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:30.516271179 +0000 UTC m=+98.045803539" watchObservedRunningTime="2026-07-06 17:53:30.518661219 +0000 UTC m=+98.048193588" Jul 06 17:53:31.303759 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:31.303709 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:32.277601 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.277564 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:32.279833 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.279812 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b71253-28c0-43c8-9e13-3f7de5bd5af8-metrics-tls\") pod \"dns-default-x88hn\" (UID: \"19b71253-28c0-43c8-9e13-3f7de5bd5af8\") " pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:32.553017 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.552995 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pm8pm\"" Jul 06 17:53:32.561394 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.561379 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:32.580835 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.580808 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:32.677220 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.677189 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:32.698268 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:32.698245 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x88hn"] Jul 06 17:53:32.700917 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:32.700890 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b71253_28c0_43c8_9e13_3f7de5bd5af8.slice/crio-65766c418602d2b76054be90860bf2b4e4b989f001a70ff2230e74da497c4278 WatchSource:0}: Error finding container 65766c418602d2b76054be90860bf2b4e4b989f001a70ff2230e74da497c4278: Status 404 returned error can't find the container with id 65766c418602d2b76054be90860bf2b4e4b989f001a70ff2230e74da497c4278 Jul 06 17:53:33.480926 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:33.480840 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x88hn" event={"ID":"19b71253-28c0-43c8-9e13-3f7de5bd5af8","Type":"ContainerStarted","Data":"65766c418602d2b76054be90860bf2b4e4b989f001a70ff2230e74da497c4278"} Jul 06 17:53:35.361163 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.361137 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s85lm" Jul 06 17:53:35.528419 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.528382 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:35.571975 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.571936 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb"] Jul 06 17:53:35.593360 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.593334 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb"] Jul 06 17:53:35.593516 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.593456 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.597420 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.597274 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 17:53:35.597420 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.597315 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 17:53:35.597658 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.597611 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 17:53:35.597803 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.597788 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 17:53:35.598141 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.598123 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-snq9x\"" Jul 06 17:53:35.598332 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.598312 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 17:53:35.601076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.600999 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474c0530-039f-4144-a0cc-ff9613a36caf-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.601076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.601067 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpl6\" (UniqueName: \"kubernetes.io/projected/474c0530-039f-4144-a0cc-ff9613a36caf-kube-api-access-4vpl6\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.601247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.601120 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.601247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.601144 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.631764 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.631673 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:35.702432 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.702402 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpl6\" (UniqueName: \"kubernetes.io/projected/474c0530-039f-4144-a0cc-ff9613a36caf-kube-api-access-4vpl6\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.702629 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.702452 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.702629 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:35.702562 2587 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 17:53:35.702629 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.702582 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.702629 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:35.702612 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls podName:474c0530-039f-4144-a0cc-ff9613a36caf nodeName:}" failed. No retries permitted until 2026-07-06 17:53:36.202596414 +0000 UTC m=+103.732128752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-q6nsb" (UID: "474c0530-039f-4144-a0cc-ff9613a36caf") : secret "prometheus-operator-tls" not found Jul 06 17:53:35.702896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.702697 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474c0530-039f-4144-a0cc-ff9613a36caf-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.703380 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.703359 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474c0530-039f-4144-a0cc-ff9613a36caf-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.705127 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.705105 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:35.715277 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:35.715258 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpl6\" (UniqueName: \"kubernetes.io/projected/474c0530-039f-4144-a0cc-ff9613a36caf-kube-api-access-4vpl6\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:36.071004 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.070968 2587 scope.go:117] "RemoveContainer" containerID="bb7f2465bc4c64eac84491cba00445955ca090d74ed7559a3232067fae9b1c57" Jul 06 17:53:36.146539 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.146491 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:36.206775 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.206735 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:36.209293 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.209272 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/474c0530-039f-4144-a0cc-ff9613a36caf-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-q6nsb\" (UID: \"474c0530-039f-4144-a0cc-ff9613a36caf\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:36.350339 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.350250 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:36.504228 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.504199 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" Jul 06 17:53:36.853618 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:36.853587 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb"] Jul 06 17:53:36.857609 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:36.857580 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474c0530_039f_4144_a0cc_ff9613a36caf.slice/crio-1ddf605631e858aea4f3b99c1f88f92d363b3fe9dd3695564eaa8279900d2ba0 WatchSource:0}: Error finding container 1ddf605631e858aea4f3b99c1f88f92d363b3fe9dd3695564eaa8279900d2ba0: Status 404 returned error can't find the container with id 1ddf605631e858aea4f3b99c1f88f92d363b3fe9dd3695564eaa8279900d2ba0 Jul 06 17:53:37.497783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.497693 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 17:53:37.497940 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.497835 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" event={"ID":"d22ec3d0-b287-4110-a439-f96f099270d3","Type":"ContainerStarted","Data":"f758023c7835b2a1f25aaf8734a1d3dd9ae7cd814743a853de82468ca6d7e5dd"} Jul 06 17:53:37.498273 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.498146 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:37.501119 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.501092 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x88hn" event={"ID":"19b71253-28c0-43c8-9e13-3f7de5bd5af8","Type":"ContainerStarted","Data":"3145c7b12e0837df1cedd42aebbf7996edd58f5c4e42ab9b7f211b6262cccbd6"} Jul 06 17:53:37.501268 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.501149 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:37.501268 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.501175 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x88hn" event={"ID":"19b71253-28c0-43c8-9e13-3f7de5bd5af8","Type":"ContainerStarted","Data":"71441aa22105bf37d106ab37a8a266d9486790f2dcae062903d5440403d41839"} Jul 06 17:53:37.502672 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.502614 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" event={"ID":"474c0530-039f-4144-a0cc-ff9613a36caf","Type":"ContainerStarted","Data":"1ddf605631e858aea4f3b99c1f88f92d363b3fe9dd3695564eaa8279900d2ba0"} Jul 06 17:53:37.504139 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.504119 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" event={"ID":"8a88d641-d614-472c-a210-aba0a2e09fc8","Type":"ContainerStarted","Data":"dee10e49f72e5729f23adee47adb3b0178d41dc3a2640781a9350cd54ad82194"} Jul 06 17:53:37.504356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.504311 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:37.506005 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.505985 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" Jul 06 17:53:37.518787 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.518705 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podStartSLOduration=45.04527592 podStartE2EDuration="47.518689523s" totalImagesPullingTime="2.473413603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:37.518386943 +0000 UTC m=+105.047919305" watchObservedRunningTime="2026-07-06 17:53:37.518689523 +0000 UTC m=+105.048221884" Jul 06 17:53:37.538301 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.538245 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x88hn" podStartSLOduration=65.521692402 podStartE2EDuration="1m9.538229171s" totalImagesPullingTime="4.016536769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:37.537642552 +0000 UTC m=+105.067174902" watchObservedRunningTime="2026-07-06 17:53:37.538229171 +0000 UTC m=+105.067761532" Jul 06 17:53:37.558677 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:37.558631 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666564b8d6-h7rqm" podStartSLOduration=1.830133276 podStartE2EDuration="8.558614152s" totalImagesPullingTime="6.728480876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:37.555435144 +0000 UTC m=+105.084967504" watchObservedRunningTime="2026-07-06 17:53:37.558614152 +0000 UTC m=+105.088146511" Jul 06 17:53:38.498211 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:38.498186 2587 patch_prober.go:28] interesting pod/console-operator-68b757fb67-lc5tx container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/readyz\": context deadline exceeded" start-of-body= Jul 06 17:53:38.498334 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:38.498242 2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" podUID="d22ec3d0-b287-4110-a439-f96f099270d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.11:8443/readyz\": context deadline exceeded" Jul 06 17:53:38.596270 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:38.596240 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-lc5tx" Jul 06 17:53:38.950913 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:38.950871 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.025420 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.025384 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.121112 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.121080 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.491160 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.491118 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.512162 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.512129 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" event={"ID":"474c0530-039f-4144-a0cc-ff9613a36caf","Type":"ContainerStarted","Data":"a934f01af6ea963589ef11cd6c42dfc54e2c035f5e3e3cb2a872c52fa1dbe6b7"} Jul 06 17:53:39.512285 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.512189 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" event={"ID":"474c0530-039f-4144-a0cc-ff9613a36caf","Type":"ContainerStarted","Data":"6aa7b20654fc15668e71596478f01e5e029550403f700f82492440e4c087e65c"} Jul 06 17:53:39.538221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.538181 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-q6nsb" podStartSLOduration=2.899342322 podStartE2EDuration="4.538169117s" totalImagesPullingTime="1.638826795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:39.53808455 +0000 UTC m=+107.067616910" watchObservedRunningTime="2026-07-06 17:53:39.538169117 +0000 UTC m=+107.067701454" Jul 06 17:53:39.585182 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.585146 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.683529 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.683496 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.763626 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.763553 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:39.907134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:39.907103 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:40.311206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:40.311174 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:40.503483 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:40.503441 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:40.695826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:40.695709 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:41.554265 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.554232 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:41.671344 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.671313 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:41.782537 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.782504 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:41.910524 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.910429 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:41.960958 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.960927 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nz9rr"] Jul 06 17:53:41.963875 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.963846 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:41.967249 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.967229 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 17:53:41.967249 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.967236 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 17:53:41.967451 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.967384 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 17:53:41.968100 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:41.968085 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r4hp5\"" Jul 06 17:53:42.001579 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.001543 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.054590 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054560 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-root\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054604 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054664 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054685 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-sys\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054740 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054764 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-wtmp\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054947 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054794 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-textfile\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054947 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054816 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v226x\" (UniqueName: \"kubernetes.io/projected/b76a5558-6c9e-4b1e-81f6-3e7060417548-kube-api-access-v226x\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.054947 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.054841 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-metrics-client-ca\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.102542 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.102505 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.155627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155590 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-root\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.155627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155632 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.155876 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155706 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-root\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.155876 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:42.155754 2587 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 17:53:42.155876 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155778 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.155876 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:42.155815 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls podName:b76a5558-6c9e-4b1e-81f6-3e7060417548 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:42.655793426 +0000 UTC m=+110.185325786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls") pod "node-exporter-nz9rr" (UID: "b76a5558-6c9e-4b1e-81f6-3e7060417548") : secret "node-exporter-tls" not found Jul 06 17:53:42.155876 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155866 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-sys\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155900 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155951 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-wtmp\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.155956 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-sys\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156007 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-textfile\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156039 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v226x\" (UniqueName: \"kubernetes.io/projected/b76a5558-6c9e-4b1e-81f6-3e7060417548-kube-api-access-v226x\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156086 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-metrics-client-ca\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156180 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-wtmp\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156550 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156529 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-textfile\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156695 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156678 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-metrics-client-ca\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.156773 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.156734 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.158262 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.158234 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.165436 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.165384 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v226x\" (UniqueName: \"kubernetes.io/projected/b76a5558-6c9e-4b1e-81f6-3e7060417548-kube-api-access-v226x\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.271502 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.271475 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.402613 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.402586 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.659749 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.659698 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.661875 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.661841 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.662185 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.662144 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b76a5558-6c9e-4b1e-81f6-3e7060417548-node-exporter-tls\") pod \"node-exporter-nz9rr\" (UID: \"b76a5558-6c9e-4b1e-81f6-3e7060417548\") " pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.758535 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.758478 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:42.872754 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.872701 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz9rr" Jul 06 17:53:42.883487 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:42.883462 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76a5558_6c9e_4b1e_81f6_3e7060417548.slice/crio-472483afc785117aa33eb58989b433ca3d0e9bcab5c8b29dc9a8d26ffd2f4c28 WatchSource:0}: Error finding container 472483afc785117aa33eb58989b433ca3d0e9bcab5c8b29dc9a8d26ffd2f4c28: Status 404 returned error can't find the container with id 472483afc785117aa33eb58989b433ca3d0e9bcab5c8b29dc9a8d26ffd2f4c28 Jul 06 17:53:42.939017 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:42.938844 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:43.038335 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.038297 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:43.039901 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.039876 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:53:43.042474 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.042454 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.044989 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.044969 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 17:53:43.045104 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.044971 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 17:53:43.045325 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045310 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 17:53:43.045425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045406 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 17:53:43.045592 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045572 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 17:53:43.045734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045613 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lckp6\"" Jul 06 17:53:43.045734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045631 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 17:53:43.045734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045613 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 17:53:43.045734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045672 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 17:53:43.045734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.045674 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 17:53:43.059474 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.059449 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:53:43.064375 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064355 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064475 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064384 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064475 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064408 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtb5l\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064475 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064466 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064484 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064505 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064533 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064572 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064640 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064628 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064665 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064700 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064740 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.064838 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.064766 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165120 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165093 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165120 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165123 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165147 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtb5l\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165187 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165203 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165222 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165239 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165255 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165277 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165300 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165327 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165352 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.165900 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.165376 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.166059 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.166031 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.166219 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.166112 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.166511 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.166490 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.168850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.168097 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.168850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.168225 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.168850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.168512 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.168850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.168582 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:43.168850 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.168767 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.169549 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.169524 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.170238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.170214 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.170349 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.170329 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.170578 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.170559 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.171055 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.171034 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.175888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.175867 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtb5l\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l\") pod \"alertmanager-main-0\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.354080 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.354039 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:53:43.494120 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.494096 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:53:43.495504 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:43.495478 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e1f05_a74a_406b_b79e_01a901a5beb4.slice/crio-f14f1f0b3c7e56ebced64a8aca970b99e72a0c3d8d8c7752f84acfa531286548 WatchSource:0}: Error finding container f14f1f0b3c7e56ebced64a8aca970b99e72a0c3d8d8c7752f84acfa531286548: Status 404 returned error can't find the container with id f14f1f0b3c7e56ebced64a8aca970b99e72a0c3d8d8c7752f84acfa531286548 Jul 06 17:53:43.526041 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.526011 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"f14f1f0b3c7e56ebced64a8aca970b99e72a0c3d8d8c7752f84acfa531286548"} Jul 06 17:53:43.527412 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:43.527387 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz9rr" event={"ID":"b76a5558-6c9e-4b1e-81f6-3e7060417548","Type":"ContainerStarted","Data":"472483afc785117aa33eb58989b433ca3d0e9bcab5c8b29dc9a8d26ffd2f4c28"} Jul 06 17:53:44.537152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:44.537116 2587 generic.go:356] "Generic (PLEG): container finished" podID="b76a5558-6c9e-4b1e-81f6-3e7060417548" containerID="15ad399b11c2217966f29db35e9f1c42988e92813993801b0969b7e289234a22" exitCode=0 Jul 06 17:53:44.537589 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:44.537184 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz9rr" event={"ID":"b76a5558-6c9e-4b1e-81f6-3e7060417548","Type":"ContainerDied","Data":"15ad399b11c2217966f29db35e9f1c42988e92813993801b0969b7e289234a22"} Jul 06 17:53:45.026438 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.026407 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:45.030390 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.030362 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fb5657dd-p5ls7"] Jul 06 17:53:45.032930 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.032909 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.035923 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.035896 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-avqvs4io72kvd\"" Jul 06 17:53:45.036068 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.035977 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 17:53:45.036068 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.036005 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 17:53:45.036068 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.036021 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 17:53:45.036068 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.035905 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 17:53:45.036266 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.036088 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6ph84\"" Jul 06 17:53:45.036376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.036362 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 17:53:45.050601 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.050576 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb5657dd-p5ls7"] Jul 06 17:53:45.082379 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082350 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082406 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082458 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082481 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zg8\" (UniqueName: \"kubernetes.io/projected/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-kube-api-access-t7zg8\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082615 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082509 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-metrics-client-ca\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082615 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082589 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082683 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082617 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-grpc-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.082683 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.082642 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.091131 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.091104 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:45.183134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183103 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183250 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183140 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-grpc-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183250 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183166 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183250 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183197 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183249 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183279 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183304 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zg8\" (UniqueName: \"kubernetes.io/projected/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-kube-api-access-t7zg8\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.183376 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.183337 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-metrics-client-ca\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.184142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.184115 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-metrics-client-ca\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186270 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186237 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186371 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186319 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-grpc-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186371 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186336 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186380 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186419 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.186593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.186564 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-secret-thanos-querier-tls\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.191003 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.190981 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zg8\" (UniqueName: \"kubernetes.io/projected/ef8f61a7-60bf-4f3c-813c-75b8ac80a75d-kube-api-access-t7zg8\") pod \"thanos-querier-fb5657dd-p5ls7\" (UID: \"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d\") " pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.195416 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.195391 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:45.342290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.342242 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:45.472116 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.472079 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb5657dd-p5ls7"] Jul 06 17:53:45.475541 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:45.475518 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8f61a7_60bf_4f3c_813c_75b8ac80a75d.slice/crio-9d7ebabb1073de60e8d5a4138692d330fe8175d1fd2f41faf33559cf1e380fda WatchSource:0}: Error finding container 9d7ebabb1073de60e8d5a4138692d330fe8175d1fd2f41faf33559cf1e380fda: Status 404 returned error can't find the container with id 9d7ebabb1073de60e8d5a4138692d330fe8175d1fd2f41faf33559cf1e380fda Jul 06 17:53:45.543011 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.542977 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz9rr" event={"ID":"b76a5558-6c9e-4b1e-81f6-3e7060417548","Type":"ContainerStarted","Data":"d3041385b11ec3581b19e62f72afdbb6d12aa5e96f9216b42dc8f0630ea6c17f"} Jul 06 17:53:45.543442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.543028 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz9rr" event={"ID":"b76a5558-6c9e-4b1e-81f6-3e7060417548","Type":"ContainerStarted","Data":"0c1726966334736ecf8324f0c9972c81587e830d20e86b0022dde42a3dcb7011"} Jul 06 17:53:45.544182 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.544161 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"9d7ebabb1073de60e8d5a4138692d330fe8175d1fd2f41faf33559cf1e380fda"} Jul 06 17:53:45.545433 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.545409 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d" exitCode=0 Jul 06 17:53:45.545524 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.545440 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d"} Jul 06 17:53:45.571083 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:45.571031 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nz9rr" podStartSLOduration=3.4358903610000002 podStartE2EDuration="4.571016059s" totalImagesPullingTime="1.07192525s" totalInitContainerRuntime="63.200448ms" isStatefulPod=true podCreationTimestamp="2026-07-06 17:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:45.570781174 +0000 UTC m=+113.100313535" watchObservedRunningTime="2026-07-06 17:53:45.571016059 +0000 UTC m=+113.100548416" Jul 06 17:53:46.064971 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.064937 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:46.721221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.721181 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:46.729674 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.729631 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np"] Jul 06 17:53:46.733825 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.733802 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:46.736488 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.736287 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 17:53:46.736488 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.736418 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4m92j\"" Jul 06 17:53:46.744421 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.744394 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np"] Jul 06 17:53:46.799094 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.799056 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:46.799287 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.799218 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfhv\" (UniqueName: \"kubernetes.io/projected/a9f4da1f-40dd-4690-9373-0eb32200b5c0-kube-api-access-qnfhv\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:46.826982 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.826942 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:46.900498 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.900460 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfhv\" (UniqueName: \"kubernetes.io/projected/a9f4da1f-40dd-4690-9373-0eb32200b5c0-kube-api-access-qnfhv\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:46.900666 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.900523 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:46.900738 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:46.900692 2587 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 17:53:46.900784 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:46.900765 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert podName:a9f4da1f-40dd-4690-9373-0eb32200b5c0 nodeName:}" failed. No retries permitted until 2026-07-06 17:53:47.400745444 +0000 UTC m=+114.930277781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-7n6np" (UID: "a9f4da1f-40dd-4690-9373-0eb32200b5c0") : secret "monitoring-plugin-cert" not found Jul 06 17:53:46.911820 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:46.911791 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfhv\" (UniqueName: \"kubernetes.io/projected/a9f4da1f-40dd-4690-9373-0eb32200b5c0-kube-api-access-qnfhv\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:47.040807 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:47.040684 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:47.405403 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:47.405359 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:47.407637 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:47.407611 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f4da1f-40dd-4690-9373-0eb32200b5c0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7n6np\" (UID: \"a9f4da1f-40dd-4690-9373-0eb32200b5c0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:47.509814 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:47.509779 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x88hn" Jul 06 17:53:47.647894 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:47.647864 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:48.141249 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.139877 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:53:48.142828 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.142769 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.145399 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.145357 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.146583 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.146832 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.147033 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.147245 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nsjbk\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.147432 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.147600 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 17:53:48.148236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.147814 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 17:53:48.154980 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.154955 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:53:48.214648 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214611 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.214852 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214674 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.214852 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214766 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.214852 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214783 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.215004 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214946 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.215004 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.214988 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnzv\" (UniqueName: \"kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.278881 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.278842 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:48.315840 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315804 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.315840 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315838 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnzv\" (UniqueName: \"kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316052 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315862 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316052 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315896 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316052 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315922 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316052 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.315936 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316863 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.316659 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.316863 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.316659 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.317186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.317014 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.319258 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.319215 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.320148 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.320103 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.325356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.325336 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnzv\" (UniqueName: \"kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv\") pod \"console-5b65bb56cb-rj9tb\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.460165 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.459959 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:48.511054 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.510967 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np"] Jul 06 17:53:48.514391 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:48.514367 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f4da1f_40dd_4690_9373_0eb32200b5c0.slice/crio-937b09526be5b2b04ec8f65118bcae42042c54244670a38823f3dccfe3195417 WatchSource:0}: Error finding container 937b09526be5b2b04ec8f65118bcae42042c54244670a38823f3dccfe3195417: Status 404 returned error can't find the container with id 937b09526be5b2b04ec8f65118bcae42042c54244670a38823f3dccfe3195417 Jul 06 17:53:48.563678 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.563579 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7"} Jul 06 17:53:48.566415 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.566297 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" event={"ID":"a9f4da1f-40dd-4690-9373-0eb32200b5c0","Type":"ContainerStarted","Data":"937b09526be5b2b04ec8f65118bcae42042c54244670a38823f3dccfe3195417"} Jul 06 17:53:48.568631 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.568565 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"acea833ee52f9ee5e5b53a06b23a9358d6a85e63fc8a2a26152b481af126f9b1"} Jul 06 17:53:48.651642 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:48.651548 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:53:48.655292 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:53:48.655247 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b377f67_4865_466a_90c5_e6f689c26532.slice/crio-f32b8438a2d610ece3132616402687c66a63ecf745b4bc7d316190d9c3b0b90d WatchSource:0}: Error finding container f32b8438a2d610ece3132616402687c66a63ecf745b4bc7d316190d9c3b0b90d: Status 404 returned error can't find the container with id f32b8438a2d610ece3132616402687c66a63ecf745b4bc7d316190d9c3b0b90d Jul 06 17:53:49.575475 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.575418 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b65bb56cb-rj9tb" event={"ID":"0b377f67-4865-466a-90c5-e6f689c26532","Type":"ContainerStarted","Data":"f32b8438a2d610ece3132616402687c66a63ecf745b4bc7d316190d9c3b0b90d"} Jul 06 17:53:49.579603 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.579502 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71"} Jul 06 17:53:49.579603 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.579577 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460"} Jul 06 17:53:49.579808 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.579615 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d"} Jul 06 17:53:49.579808 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.579651 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48"} Jul 06 17:53:49.582658 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.582596 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"872744336cef77c46ff1c44b9ffbf211578407b7bd71ee645e8c1e463b7f9c31"} Jul 06 17:53:49.582800 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.582671 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"e3685c5e2627b8c34f79a0456bef74540f182ac185419586041f39000abd2055"} Jul 06 17:53:49.888400 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:49.888284 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:50.145929 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.145888 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:50.264128 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.264097 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:50.591441 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.591400 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerStarted","Data":"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b"} Jul 06 17:53:50.593244 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.593208 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" event={"ID":"a9f4da1f-40dd-4690-9373-0eb32200b5c0","Type":"ContainerStarted","Data":"acc2e07f652a382e50390f949332fe1a4eb2ab5e78525506f5291dab6914df18"} Jul 06 17:53:50.593365 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.593354 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:50.596795 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.596760 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"6b6286b1d7327be3b9bfa872cd1403e80d38bbd60db22f4b579b83a66d699e6c"} Jul 06 17:53:50.596944 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.596848 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"fb17f2b8b758f97bd655d66dfde20a9881600537b87b3fd5c1ff30f5bbd4f19f"} Jul 06 17:53:50.596944 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.596896 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" event={"ID":"ef8f61a7-60bf-4f3c-813c-75b8ac80a75d","Type":"ContainerStarted","Data":"eccb5d06b77635341a4c3a5713bd491e2e9d1d44f93e60c3ce678b7f34887059"} Jul 06 17:53:50.597111 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.597084 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:50.599956 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.599935 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" Jul 06 17:53:50.622671 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.622626 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.25241392 podStartE2EDuration="7.622612098s" totalImagesPullingTime="5.352575572s" totalInitContainerRuntime="17.622606ms" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:50.620612046 +0000 UTC m=+118.150144408" watchObservedRunningTime="2026-07-06 17:53:50.622612098 +0000 UTC m=+118.152144457" Jul 06 17:53:50.641055 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.641002 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7n6np" podStartSLOduration=3.065668702 podStartE2EDuration="4.640981044s" totalImagesPullingTime="1.575312342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:50.639472863 +0000 UTC m=+118.169005224" watchObservedRunningTime="2026-07-06 17:53:50.640981044 +0000 UTC m=+118.170513410" Jul 06 17:53:50.663321 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:50.663275 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" podStartSLOduration=1.334233723 podStartE2EDuration="5.663257744s" totalImagesPullingTime="4.329024021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:50.662670551 +0000 UTC m=+118.192202931" watchObservedRunningTime="2026-07-06 17:53:50.663257744 +0000 UTC m=+118.192790104" Jul 06 17:53:52.607809 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.607764 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b65bb56cb-rj9tb" event={"ID":"0b377f67-4865-466a-90c5-e6f689c26532","Type":"ContainerStarted","Data":"73ebb0fc0a71aa64ddd439ed08024a56583e7d284ac0ca05143013d6e1e2510f"} Jul 06 17:53:52.630572 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.630522 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b65bb56cb-rj9tb" podStartSLOduration=1.197206501 podStartE2EDuration="4.630507564s" totalImagesPullingTime="3.433301063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:53:52.628459075 +0000 UTC m=+120.157991435" watchObservedRunningTime="2026-07-06 17:53:52.630507564 +0000 UTC m=+120.160039927" Jul 06 17:53:52.635654 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.635621 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:52.711582 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.711546 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:52.808406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.808371 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:52.938835 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:52.938747 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:53.615360 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:53.615323 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:53.671679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:53.671650 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:54.909072 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:54.909034 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" podUID="3cae64d9-0470-4dec-98d3-ec42f366fefb" containerName="registry" containerID="cri-o://2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027" gracePeriod=30 Jul 06 17:53:55.150404 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.150381 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:55.281935 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.281858 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.281935 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.281908 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.281935 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.281934 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.281967 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdczd\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282014 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282044 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282064 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282079 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates\") pod \"3cae64d9-0470-4dec-98d3-ec42f366fefb\" (UID: \"3cae64d9-0470-4dec-98d3-ec42f366fefb\") " Jul 06 17:53:55.282764 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282736 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:53:55.282922 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.282840 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:53:55.284854 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.284826 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:53:55.284981 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.284872 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd" (OuterVolumeSpecName: "kube-api-access-wdczd") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "kube-api-access-wdczd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:53:55.284981 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.284944 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:53:55.285076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.285032 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:53:55.285076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.285045 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:53:55.290474 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.290448 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3cae64d9-0470-4dec-98d3-ec42f366fefb" (UID: "3cae64d9-0470-4dec-98d3-ec42f366fefb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:53:55.382681 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382649 2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-trusted-ca\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382681 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382681 2587 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-image-registry-private-configuration\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382694 2587 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3cae64d9-0470-4dec-98d3-ec42f366fefb-installation-pull-secrets\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382704 2587 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-certificates\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382731 2587 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-registry-tls\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382740 2587 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-bound-sa-token\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382751 2587 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3cae64d9-0470-4dec-98d3-ec42f366fefb-ca-trust-extracted\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.382896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.382763 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdczd\" (UniqueName: \"kubernetes.io/projected/3cae64d9-0470-4dec-98d3-ec42f366fefb-kube-api-access-wdczd\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:53:55.526100 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.526066 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:55.621971 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.621935 2587 generic.go:356] "Generic (PLEG): container finished" podID="3cae64d9-0470-4dec-98d3-ec42f366fefb" containerID="2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027" exitCode=0 Jul 06 17:53:55.622147 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.622007 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" Jul 06 17:53:55.622147 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.622022 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" event={"ID":"3cae64d9-0470-4dec-98d3-ec42f366fefb","Type":"ContainerDied","Data":"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027"} Jul 06 17:53:55.622147 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.622108 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868bb7f4b8-7gmdw" event={"ID":"3cae64d9-0470-4dec-98d3-ec42f366fefb","Type":"ContainerDied","Data":"66e4d38b0388bd83e049494556034142ec64c878b9b595eec68b42158ea1a368"} Jul 06 17:53:55.622147 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.622134 2587 scope.go:117] "RemoveContainer" containerID="2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027" Jul 06 17:53:55.627622 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.627592 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:55.630739 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.630707 2587 scope.go:117] "RemoveContainer" containerID="2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027" Jul 06 17:53:55.630993 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:53:55.630973 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027\": container with ID starting with 2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027 not found: ID does not exist" containerID="2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027" Jul 06 17:53:55.631041 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.631001 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027"} err="failed to get container status \"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027\": rpc error: code = NotFound desc = could not find container \"2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027\": container with ID starting with 2e249f61dd0cc6940d30eb49c59fee041811e223abf805cee20b115e1dd51027 not found: ID does not exist" Jul 06 17:53:55.644148 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.644121 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:53:55.648573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:55.648552 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-868bb7f4b8-7gmdw"] Jul 06 17:53:56.607520 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:56.607497 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fb5657dd-p5ls7" Jul 06 17:53:57.076323 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:57.076293 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cae64d9-0470-4dec-98d3-ec42f366fefb" path="/var/lib/kubelet/pods/3cae64d9-0470-4dec-98d3-ec42f366fefb/volumes" Jul 06 17:53:58.407086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.407056 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:58.461057 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.461027 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:58.461193 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.461094 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:58.465936 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.465912 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:53:58.506057 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.506019 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:53:58.638675 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:53:58.638647 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:54:00.716747 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:00.716693 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:02.633159 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.633127 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:02.741334 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.741297 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:02.746695 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.746669 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:54:02.748894 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.748874 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051b52de-5f8c-4f51-88fb-0b286ab70161-metrics-certs\") pod \"network-metrics-daemon-c955d\" (UID: \"051b52de-5f8c-4f51-88fb-0b286ab70161\") " pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:54:02.800132 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.800106 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-22ww9\"" Jul 06 17:54:02.806698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.806677 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c955d" Jul 06 17:54:02.969268 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:02.969146 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c955d"] Jul 06 17:54:02.971381 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:54:02.971353 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051b52de_5f8c_4f51_88fb_0b286ab70161.slice/crio-18f6b94fd0fee968f795c0e283752cea4fd7612e616263f50bc27d93a57d775f WatchSource:0}: Error finding container 18f6b94fd0fee968f795c0e283752cea4fd7612e616263f50bc27d93a57d775f: Status 404 returned error can't find the container with id 18f6b94fd0fee968f795c0e283752cea4fd7612e616263f50bc27d93a57d775f Jul 06 17:54:03.656282 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:03.656249 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c955d" event={"ID":"051b52de-5f8c-4f51-88fb-0b286ab70161","Type":"ContainerStarted","Data":"18f6b94fd0fee968f795c0e283752cea4fd7612e616263f50bc27d93a57d775f"} Jul 06 17:54:03.892993 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:03.892938 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:03.998120 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:03.998009 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:04.101605 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.101573 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:04.662301 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.662267 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c955d" event={"ID":"051b52de-5f8c-4f51-88fb-0b286ab70161","Type":"ContainerStarted","Data":"559a7ddc0056ffd840ba4b94d36ec44be3e92eabcb84d047ee1aec86ab1bbbed"} Jul 06 17:54:04.662697 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.662368 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c955d" event={"ID":"051b52de-5f8c-4f51-88fb-0b286ab70161","Type":"ContainerStarted","Data":"b72beeb00fbd5eb258d5d1b84b8913e76c5c015877745f632cf4f95b2ca9c0ce"} Jul 06 17:54:04.686102 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.686059 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c955d" podStartSLOduration=130.525092707 podStartE2EDuration="2m11.686044151s" totalImagesPullingTime="1.160951444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:54:04.684979407 +0000 UTC m=+132.214511767" watchObservedRunningTime="2026-07-06 17:54:04.686044151 +0000 UTC m=+132.215576507" Jul 06 17:54:04.769360 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.769332 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:04.869396 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.869361 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:04.937800 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:04.937701 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:05.133442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:05.133405 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:05.233615 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:05.233538 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:06.544125 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:06.544092 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:54:06.691688 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:06.691656 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:06.792128 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:06.792090 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:06.891605 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:06.891552 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:06.993491 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:06.993462 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:07.310356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:07.310325 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:07.382902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:07.382861 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:10.686401 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:10.686368 2587 generic.go:356] "Generic (PLEG): container finished" podID="01cb4dd5-3f35-4e37-8b28-a849b62b9e70" containerID="f727fd0f7a9436fb29b49178b8fe3f52ffbff6ad94b273ed40b46fa23e26a755" exitCode=0 Jul 06 17:54:10.686783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:10.686437 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" event={"ID":"01cb4dd5-3f35-4e37-8b28-a849b62b9e70","Type":"ContainerDied","Data":"f727fd0f7a9436fb29b49178b8fe3f52ffbff6ad94b273ed40b46fa23e26a755"} Jul 06 17:54:10.686848 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:10.686797 2587 scope.go:117] "RemoveContainer" containerID="f727fd0f7a9436fb29b49178b8fe3f52ffbff6ad94b273ed40b46fa23e26a755" Jul 06 17:54:11.564273 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:11.564239 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-2rh58_bc6bac78-03a2-4366-8df3-a331874e9d94/cluster-samples-operator/0.log" Jul 06 17:54:11.570797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:11.570768 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-2rh58_bc6bac78-03a2-4366-8df3-a331874e9d94/cluster-samples-operator-watch/0.log" Jul 06 17:54:11.685722 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:11.685677 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:11.692290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:11.692264 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jj9mh" event={"ID":"01cb4dd5-3f35-4e37-8b28-a849b62b9e70","Type":"ContainerStarted","Data":"41f6b8a76d4844ca943999f2a3f29a17d2e5bfba38e9a46c59d81d29031abc31"} Jul 06 17:54:11.785175 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:11.785145 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:16.399154 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:16.399117 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:16.494954 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:16.494915 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:26.683307 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:26.683265 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:26.784150 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:26.784119 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:27.685797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:27.685765 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:27.749895 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:27.749860 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:31.566192 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.566129 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5b65bb56cb-rj9tb" podUID="0b377f67-4865-466a-90c5-e6f689c26532" containerName="console" containerID="cri-o://73ebb0fc0a71aa64ddd439ed08024a56583e7d284ac0ca05143013d6e1e2510f" gracePeriod=15 Jul 06 17:54:31.772444 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.772409 2587 generic.go:356] "Generic (PLEG): container finished" podID="0b377f67-4865-466a-90c5-e6f689c26532" containerID="73ebb0fc0a71aa64ddd439ed08024a56583e7d284ac0ca05143013d6e1e2510f" exitCode=0 Jul 06 17:54:31.772617 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.772478 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b65bb56cb-rj9tb" event={"ID":"0b377f67-4865-466a-90c5-e6f689c26532","Type":"ContainerDied","Data":"73ebb0fc0a71aa64ddd439ed08024a56583e7d284ac0ca05143013d6e1e2510f"} Jul 06 17:54:31.808684 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.808662 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:54:31.899247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899167 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899198 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899247 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899232 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899252 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899270 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbnzv\" (UniqueName: \"kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899299 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config\") pod \"0b377f67-4865-466a-90c5-e6f689c26532\" (UID: \"0b377f67-4865-466a-90c5-e6f689c26532\") " Jul 06 17:54:31.899675 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899652 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config" (OuterVolumeSpecName: "console-config") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:54:31.899745 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899663 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:54:31.899745 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.899671 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca" (OuterVolumeSpecName: "service-ca") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:54:31.901604 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.901579 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:54:31.901690 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.901601 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:54:31.901690 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:31.901639 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv" (OuterVolumeSpecName: "kube-api-access-dbnzv") pod "0b377f67-4865-466a-90c5-e6f689c26532" (UID: "0b377f67-4865-466a-90c5-e6f689c26532"). InnerVolumeSpecName "kube-api-access-dbnzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:54:32.000625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000592 2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-console-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.000625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000618 2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-service-ca\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.000625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000627 2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b377f67-4865-466a-90c5-e6f689c26532-oauth-serving-cert\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.000625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000636 2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-serving-cert\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.000917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000645 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnzv\" (UniqueName: \"kubernetes.io/projected/0b377f67-4865-466a-90c5-e6f689c26532-kube-api-access-dbnzv\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.000917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.000654 2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b377f67-4865-466a-90c5-e6f689c26532-console-oauth-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:54:32.777828 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.777797 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b65bb56cb-rj9tb" event={"ID":"0b377f67-4865-466a-90c5-e6f689c26532","Type":"ContainerDied","Data":"f32b8438a2d610ece3132616402687c66a63ecf745b4bc7d316190d9c3b0b90d"} Jul 06 17:54:32.778284 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.777857 2587 scope.go:117] "RemoveContainer" containerID="73ebb0fc0a71aa64ddd439ed08024a56583e7d284ac0ca05143013d6e1e2510f" Jul 06 17:54:32.778284 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.777803 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b65bb56cb-rj9tb" Jul 06 17:54:32.801632 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.801605 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:54:32.803783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:32.803761 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b65bb56cb-rj9tb"] Jul 06 17:54:33.075146 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:33.075118 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b377f67-4865-466a-90c5-e6f689c26532" path="/var/lib/kubelet/pods/0b377f67-4865-466a-90c5-e6f689c26532/volumes" Jul 06 17:54:58.145625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:58.145592 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:54:58.217150 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:54:58.217123 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:02.472575 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.472533 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:02.473066 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.472967 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="alertmanager" containerID="cri-o://de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" gracePeriod=120 Jul 06 17:55:02.473133 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.473068 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy" containerID="cri-o://d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" gracePeriod=120 Jul 06 17:55:02.473133 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.473096 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="config-reloader" containerID="cri-o://d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" gracePeriod=120 Jul 06 17:55:02.473317 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.473076 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="prom-label-proxy" containerID="cri-o://13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" gracePeriod=120 Jul 06 17:55:02.473317 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.473068 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-web" containerID="cri-o://04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" gracePeriod=120 Jul 06 17:55:02.473317 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.473035 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-metric" containerID="cri-o://008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" gracePeriod=120 Jul 06 17:55:02.894045 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894015 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" exitCode=0 Jul 06 17:55:02.894045 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894038 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" exitCode=0 Jul 06 17:55:02.894045 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894043 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" exitCode=0 Jul 06 17:55:02.894045 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894050 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" exitCode=0 Jul 06 17:55:02.894290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894081 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b"} Jul 06 17:55:02.894290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894143 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460"} Jul 06 17:55:02.894290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894169 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48"} Jul 06 17:55:02.894290 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:02.894194 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7"} Jul 06 17:55:03.715411 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.715388 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:03.858488 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858447 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858513 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858557 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858584 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858618 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858652 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtb5l\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.858698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858687 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858753 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858784 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858819 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858861 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858894 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858947 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web\") pod \"652e1f05-a74a-406b-b79e-01a901a5beb4\" (UID: \"652e1f05-a74a-406b-b79e-01a901a5beb4\") " Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.858990 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:55:03.859037 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.859011 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:55:03.859421 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.859185 2587 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.859421 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.859208 2587 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-alertmanager-main-db\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.859875 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.859835 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:55:03.861749 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.861725 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.862200 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.862179 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out" (OuterVolumeSpecName: "config-out") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:55:03.862337 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.862175 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:55:03.862447 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.862198 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.862447 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.862360 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.862810 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.862783 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.863483 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.863461 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume" (OuterVolumeSpecName: "config-volume") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.863755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.863704 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l" (OuterVolumeSpecName: "kube-api-access-mtb5l") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "kube-api-access-mtb5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:55:03.866171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.866102 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.872205 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.872183 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config" (OuterVolumeSpecName: "web-config") pod "652e1f05-a74a-406b-b79e-01a901a5beb4" (UID: "652e1f05-a74a-406b-b79e-01a901a5beb4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:55:03.900923 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.900892 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" exitCode=0 Jul 06 17:55:03.900923 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.900917 2587 generic.go:356] "Generic (PLEG): container finished" podID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerID="04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" exitCode=0 Jul 06 17:55:03.901130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.900964 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:03.901130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.900975 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71"} Jul 06 17:55:03.901130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.901046 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d"} Jul 06 17:55:03.901130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.901072 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"652e1f05-a74a-406b-b79e-01a901a5beb4","Type":"ContainerDied","Data":"f14f1f0b3c7e56ebced64a8aca970b99e72a0c3d8d8c7752f84acfa531286548"} Jul 06 17:55:03.901130 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.901107 2587 scope.go:117] "RemoveContainer" containerID="13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" Jul 06 17:55:03.909076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.909058 2587 scope.go:117] "RemoveContainer" containerID="008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" Jul 06 17:55:03.916171 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.916155 2587 scope.go:117] "RemoveContainer" containerID="d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" Jul 06 17:55:03.922848 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.922827 2587 scope.go:117] "RemoveContainer" containerID="04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" Jul 06 17:55:03.924820 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.924799 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:03.930423 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.930405 2587 scope.go:117] "RemoveContainer" containerID="d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" Jul 06 17:55:03.930868 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.930852 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:03.937278 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.937260 2587 scope.go:117] "RemoveContainer" containerID="de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" Jul 06 17:55:03.944105 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.944089 2587 scope.go:117] "RemoveContainer" containerID="403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d" Jul 06 17:55:03.950593 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.950572 2587 scope.go:117] "RemoveContainer" containerID="13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" Jul 06 17:55:03.950891 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.950865 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b\": container with ID starting with 13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b not found: ID does not exist" containerID="13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" Jul 06 17:55:03.950984 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.950901 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b"} err="failed to get container status \"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b\": rpc error: code = NotFound desc = could not find container \"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b\": container with ID starting with 13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b not found: ID does not exist" Jul 06 17:55:03.950984 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.950925 2587 scope.go:117] "RemoveContainer" containerID="008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" Jul 06 17:55:03.951158 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.951143 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71\": container with ID starting with 008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71 not found: ID does not exist" containerID="008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" Jul 06 17:55:03.951202 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951163 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71"} err="failed to get container status \"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71\": rpc error: code = NotFound desc = could not find container \"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71\": container with ID starting with 008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71 not found: ID does not exist" Jul 06 17:55:03.951202 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951177 2587 scope.go:117] "RemoveContainer" containerID="d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" Jul 06 17:55:03.951386 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.951369 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460\": container with ID starting with d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460 not found: ID does not exist" containerID="d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" Jul 06 17:55:03.951426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951393 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460"} err="failed to get container status \"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460\": rpc error: code = NotFound desc = could not find container \"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460\": container with ID starting with d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460 not found: ID does not exist" Jul 06 17:55:03.951426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951411 2587 scope.go:117] "RemoveContainer" containerID="04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" Jul 06 17:55:03.951645 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.951629 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d\": container with ID starting with 04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d not found: ID does not exist" containerID="04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" Jul 06 17:55:03.951683 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951652 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d"} err="failed to get container status \"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d\": rpc error: code = NotFound desc = could not find container \"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d\": container with ID starting with 04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d not found: ID does not exist" Jul 06 17:55:03.951683 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951666 2587 scope.go:117] "RemoveContainer" containerID="d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" Jul 06 17:55:03.951904 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.951890 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48\": container with ID starting with d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48 not found: ID does not exist" containerID="d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" Jul 06 17:55:03.951946 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951909 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48"} err="failed to get container status \"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48\": rpc error: code = NotFound desc = could not find container \"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48\": container with ID starting with d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48 not found: ID does not exist" Jul 06 17:55:03.951946 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.951921 2587 scope.go:117] "RemoveContainer" containerID="de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" Jul 06 17:55:03.952144 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.952129 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7\": container with ID starting with de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7 not found: ID does not exist" containerID="de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" Jul 06 17:55:03.952186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952152 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7"} err="failed to get container status \"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7\": rpc error: code = NotFound desc = could not find container \"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7\": container with ID starting with de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7 not found: ID does not exist" Jul 06 17:55:03.952186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952165 2587 scope.go:117] "RemoveContainer" containerID="403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d" Jul 06 17:55:03.952365 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:55:03.952350 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d\": container with ID starting with 403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d not found: ID does not exist" containerID="403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d" Jul 06 17:55:03.952406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952369 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d"} err="failed to get container status \"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d\": rpc error: code = NotFound desc = could not find container \"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d\": container with ID starting with 403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d not found: ID does not exist" Jul 06 17:55:03.952406 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952382 2587 scope.go:117] "RemoveContainer" containerID="13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b" Jul 06 17:55:03.952583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952567 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b"} err="failed to get container status \"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b\": rpc error: code = NotFound desc = could not find container \"13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b\": container with ID starting with 13fa448cd26bdfe16fbfbe17f012d12a1d873fee991baad5f2da05f42868e57b not found: ID does not exist" Jul 06 17:55:03.952621 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952583 2587 scope.go:117] "RemoveContainer" containerID="008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71" Jul 06 17:55:03.952837 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952810 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71"} err="failed to get container status \"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71\": rpc error: code = NotFound desc = could not find container \"008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71\": container with ID starting with 008a613a99c6b8f95cead56851be7b108ef99aebe842ff496473de079a697b71 not found: ID does not exist" Jul 06 17:55:03.952895 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.952835 2587 scope.go:117] "RemoveContainer" containerID="d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460" Jul 06 17:55:03.953050 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953033 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460"} err="failed to get container status \"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460\": rpc error: code = NotFound desc = could not find container \"d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460\": container with ID starting with d8c44c3a5a1a8f3fcd4fc24e5e874accb656551239d185109bb84d83f58e0460 not found: ID does not exist" Jul 06 17:55:03.953093 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953051 2587 scope.go:117] "RemoveContainer" containerID="04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d" Jul 06 17:55:03.953256 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953239 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d"} err="failed to get container status \"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d\": rpc error: code = NotFound desc = could not find container \"04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d\": container with ID starting with 04f1bded8249d27478714171ca944c222e8afa58c28cf5e7bd6f210052db4e9d not found: ID does not exist" Jul 06 17:55:03.953294 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953258 2587 scope.go:117] "RemoveContainer" containerID="d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48" Jul 06 17:55:03.953478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953460 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48"} err="failed to get container status \"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48\": rpc error: code = NotFound desc = could not find container \"d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48\": container with ID starting with d3871a32525206ef0c9b54c40de8d19cd6a17ac64ac4cb2d696c62b3d78fbc48 not found: ID does not exist" Jul 06 17:55:03.953478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953477 2587 scope.go:117] "RemoveContainer" containerID="de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7" Jul 06 17:55:03.953751 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953695 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7"} err="failed to get container status \"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7\": rpc error: code = NotFound desc = could not find container \"de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7\": container with ID starting with de72d7f2a1c17b9374282737ca12cf4a0f2c77f18f8c2a0e94ee86cabaa4c4a7 not found: ID does not exist" Jul 06 17:55:03.953800 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953745 2587 scope.go:117] "RemoveContainer" containerID="403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d" Jul 06 17:55:03.953988 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.953970 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d"} err="failed to get container status \"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d\": rpc error: code = NotFound desc = could not find container \"403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d\": container with ID starting with 403d74033fd7cb2400be7c85074629059b9b6a60905f4ebe9b4ae56bc5103d0d not found: ID does not exist" Jul 06 17:55:03.960074 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960051 2587 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652e1f05-a74a-406b-b79e-01a901a5beb4-metrics-client-ca\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960079 2587 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-web-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960093 2587 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-tls-assets\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960106 2587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960123 2587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-main-tls\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960135 2587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960148 2587 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-cluster-tls-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960163 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtb5l\" (UniqueName: \"kubernetes.io/projected/652e1f05-a74a-406b-b79e-01a901a5beb4-kube-api-access-mtb5l\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960173 2587 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/652e1f05-a74a-406b-b79e-01a901a5beb4-config-out\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960181 2587 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-config-volume\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.960189 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.960189 2587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/652e1f05-a74a-406b-b79e-01a901a5beb4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:55:03.961803 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.961786 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:03.962046 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962034 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b377f67-4865-466a-90c5-e6f689c26532" containerName="console" Jul 06 17:55:03.962086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962047 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b377f67-4865-466a-90c5-e6f689c26532" containerName="console" Jul 06 17:55:03.962086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962062 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="init-config-reloader" Jul 06 17:55:03.962086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962070 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="init-config-reloader" Jul 06 17:55:03.962086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962077 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="prom-label-proxy" Jul 06 17:55:03.962086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962082 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="prom-label-proxy" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962090 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-web" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962097 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-web" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962105 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cae64d9-0470-4dec-98d3-ec42f366fefb" containerName="registry" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962109 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cae64d9-0470-4dec-98d3-ec42f366fefb" containerName="registry" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962117 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="alertmanager" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962122 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="alertmanager" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962127 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962132 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962139 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-metric" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962144 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-metric" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962151 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="config-reloader" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962156 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="config-reloader" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962198 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="prom-label-proxy" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962206 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-metric" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962211 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cae64d9-0470-4dec-98d3-ec42f366fefb" containerName="registry" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962217 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy-web" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962222 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="alertmanager" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962227 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="config-reloader" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962232 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b377f67-4865-466a-90c5-e6f689c26532" containerName="console" Jul 06 17:55:03.962230 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.962239 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" containerName="kube-rbac-proxy" Jul 06 17:55:03.965470 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.965455 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:03.968221 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968201 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 17:55:03.968313 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968218 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 17:55:03.968313 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968237 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lckp6\"" Jul 06 17:55:03.968464 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968447 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 17:55:03.968578 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968558 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 17:55:03.968685 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968669 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 17:55:03.968794 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968778 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 17:55:03.968794 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968781 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 17:55:03.968917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.968903 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 17:55:03.975527 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.975503 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 17:55:03.983753 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:03.983731 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:04.061448 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061415 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-config-out\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061448 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061451 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061483 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061507 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-web-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061523 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061542 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfssm\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-kube-api-access-rfssm\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061568 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061688 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061754 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061807 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061870 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061900 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.061937 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.061927 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.162834 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162763 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.162834 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162801 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162837 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162872 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162894 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.162927 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163006 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-config-out\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163054 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163087 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163116 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-web-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163138 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163154 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfssm\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-kube-api-access-rfssm\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163178 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.163809 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163782 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.164020 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.163994 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.164280 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.164258 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.165941 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.165916 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166099 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166135 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166154 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166168 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3918fb76-6fb7-49ab-817a-a68283e0298d-config-out\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166181 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166571 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166552 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-web-config\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.166979 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.166962 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.167810 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.167795 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3918fb76-6fb7-49ab-817a-a68283e0298d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.173331 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.173312 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfssm\" (UniqueName: \"kubernetes.io/projected/3918fb76-6fb7-49ab-817a-a68283e0298d-kube-api-access-rfssm\") pod \"alertmanager-main-0\" (UID: \"3918fb76-6fb7-49ab-817a-a68283e0298d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.276583 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.276554 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 17:55:04.417655 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.417628 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 17:55:04.419780 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:55:04.419752 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3918fb76_6fb7_49ab_817a_a68283e0298d.slice/crio-bc259f1933c0583ce003d01aeaff1686b31d586bdccd01c8a110326b9da3e5f0 WatchSource:0}: Error finding container bc259f1933c0583ce003d01aeaff1686b31d586bdccd01c8a110326b9da3e5f0: Status 404 returned error can't find the container with id bc259f1933c0583ce003d01aeaff1686b31d586bdccd01c8a110326b9da3e5f0 Jul 06 17:55:04.905572 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.905538 2587 generic.go:356] "Generic (PLEG): container finished" podID="3918fb76-6fb7-49ab-817a-a68283e0298d" containerID="f7bdf7bc6e849aa1818491fd877e458f1877d77e73e2b002c6ba2840f941a228" exitCode=0 Jul 06 17:55:04.906062 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.905620 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerDied","Data":"f7bdf7bc6e849aa1818491fd877e458f1877d77e73e2b002c6ba2840f941a228"} Jul 06 17:55:04.906062 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:04.905745 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"bc259f1933c0583ce003d01aeaff1686b31d586bdccd01c8a110326b9da3e5f0"} Jul 06 17:55:05.075483 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.075457 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652e1f05-a74a-406b-b79e-01a901a5beb4" path="/var/lib/kubelet/pods/652e1f05-a74a-406b-b79e-01a901a5beb4/volumes" Jul 06 17:55:05.914220 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914192 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"34b8fcffd0385a727333d5e01a4f92429c1e4acf69bdcc68756964f92f2fb640"} Jul 06 17:55:05.914585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914258 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"03032cf1db48cb34e26a51b1d0538310261ea25e8b9f09b3a2c3ba339337fc09"} Jul 06 17:55:05.914585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914287 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"68549d1bff9fa5328eb047d3a72d7b53b8810dc833f4ca092201dfa253a4bdaa"} Jul 06 17:55:05.914585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914309 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"f98fb07ec8dde3cea1abc5d2accd18299d27f9f98e492311fe954ad17c18274d"} Jul 06 17:55:05.914585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914339 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"e78e4d9f1beb5291cfadb0b19156f5a91606899154b66762b635ac4428f39509"} Jul 06 17:55:05.914585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.914364 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3918fb76-6fb7-49ab-817a-a68283e0298d","Type":"ContainerStarted","Data":"522a0fe604bc3dee7e84e18f816094316a45bcc3c656245173fe5b8fe467459d"} Jul 06 17:55:05.944953 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:05.944904 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.925951083 podStartE2EDuration="2.944887179s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.936096ms" isStatefulPod=false podCreationTimestamp="2026-07-06 17:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:55:05.943353804 +0000 UTC m=+193.472886165" watchObservedRunningTime="2026-07-06 17:55:05.944887179 +0000 UTC m=+193.474419539" Jul 06 17:55:10.054893 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:10.054859 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:10.119888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:10.119843 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:26.698725 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:26.698675 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:26.797355 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:26.797321 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:26.897584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:26.897547 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:28.578099 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.578068 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:55:28.581592 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.581574 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.586344 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.586326 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 17:55:28.586782 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.586761 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 17:55:28.586939 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.586911 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 17:55:28.587102 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.587015 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nsjbk\"" Jul 06 17:55:28.587504 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.587484 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 17:55:28.587602 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.587506 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 17:55:28.587602 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.587597 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 17:55:28.587994 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.587979 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 17:55:28.591040 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.591024 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 17:55:28.615988 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.615965 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:55:28.652219 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.652185 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:28.660431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660401 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660551 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660435 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660551 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660467 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660551 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660532 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660680 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660601 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mns\" (UniqueName: \"kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660680 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660627 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.660680 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.660641 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.736647 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.736612 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:28.761392 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761368 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761396 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761518 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761418 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761596 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761537 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761655 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761638 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mns\" (UniqueName: \"kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761745 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761689 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.761800 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.761745 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.762235 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.762210 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.762436 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.762408 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.762532 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.762484 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.762794 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.762776 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.764508 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.764487 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.764585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.764509 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.772686 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.772664 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mns\" (UniqueName: \"kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns\") pod \"console-74c5d95fb-6rmt4\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:28.890920 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:28.890835 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:29.034906 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:29.034878 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:55:29.037787 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:55:29.037763 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08833a90_f880_4741_9da1_81724ae1dbfe.slice/crio-9b32989a94c08ecb0d98490d4453f6454580e1384c311db81c7c5068683a2413 WatchSource:0}: Error finding container 9b32989a94c08ecb0d98490d4453f6454580e1384c311db81c7c5068683a2413: Status 404 returned error can't find the container with id 9b32989a94c08ecb0d98490d4453f6454580e1384c311db81c7c5068683a2413 Jul 06 17:55:29.656487 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:29.656456 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:30.006126 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:30.006042 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c5d95fb-6rmt4" event={"ID":"08833a90-f880-4741-9da1-81724ae1dbfe","Type":"ContainerStarted","Data":"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66"} Jul 06 17:55:30.006126 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:30.006091 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c5d95fb-6rmt4" event={"ID":"08833a90-f880-4741-9da1-81724ae1dbfe","Type":"ContainerStarted","Data":"9b32989a94c08ecb0d98490d4453f6454580e1384c311db81c7c5068683a2413"} Jul 06 17:55:30.026954 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:30.026903 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c5d95fb-6rmt4" podStartSLOduration=2.026886208 podStartE2EDuration="2.026886208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:55:30.025962314 +0000 UTC m=+217.555494698" watchObservedRunningTime="2026-07-06 17:55:30.026886208 +0000 UTC m=+217.556418567" Jul 06 17:55:38.891101 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:38.891073 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:38.891484 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:38.891118 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:38.896766 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:38.896744 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:39.045522 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:39.045492 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:55:39.271900 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:39.271801 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:39.338685 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:39.338655 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:39.439980 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:39.439942 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:39.513164 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:39.513131 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:41.628186 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:41.628155 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:55:41.687215 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:55:41.687175 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:09.549753 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:09.549685 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:09.648254 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:09.648220 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:10.771957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.771922 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc"] Jul 06 17:56:10.776472 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.776454 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:10.784252 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.784230 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 17:56:10.784985 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.784968 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m9qjl\"" Jul 06 17:56:10.785353 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.785339 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 17:56:10.790478 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.790457 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc"] Jul 06 17:56:10.916419 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.916391 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:10.916597 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.916448 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:10.916597 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:10.916506 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.017197 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.017161 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.017197 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.017200 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.017410 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.017254 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.017623 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.017600 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.017623 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.017614 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.027348 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.027298 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.085442 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.085418 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:11.220074 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:11.219909 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc"] Jul 06 17:56:11.222296 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:11.222257 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9163efdc_c060_4603_aabf_985bf36aeb47.slice/crio-8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57 WatchSource:0}: Error finding container 8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57: Status 404 returned error can't find the container with id 8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57 Jul 06 17:56:12.168006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:12.167969 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerStarted","Data":"8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57"} Jul 06 17:56:20.201771 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:20.201740 2587 generic.go:356] "Generic (PLEG): container finished" podID="9163efdc-c060-4603-aabf-985bf36aeb47" containerID="b6dc19abca031346341798e2a710d56a27053761625ca913293e5a03b52491db" exitCode=0 Jul 06 17:56:20.202152 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:20.201820 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerDied","Data":"b6dc19abca031346341798e2a710d56a27053761625ca913293e5a03b52491db"} Jul 06 17:56:22.213929 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:22.213893 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerStarted","Data":"839eb2bb24ce7917307c3f54a8278d4c04021fe79f0d89b8da88892a54741b89"} Jul 06 17:56:23.219826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:23.219789 2587 generic.go:356] "Generic (PLEG): container finished" podID="9163efdc-c060-4603-aabf-985bf36aeb47" containerID="839eb2bb24ce7917307c3f54a8278d4c04021fe79f0d89b8da88892a54741b89" exitCode=0 Jul 06 17:56:23.220228 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:23.219888 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerDied","Data":"839eb2bb24ce7917307c3f54a8278d4c04021fe79f0d89b8da88892a54741b89"} Jul 06 17:56:32.257890 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:32.257853 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerStarted","Data":"e895e2ffec2be666cd4ad0de64db11629bf21b48ba0e25d0f22e349f6b698bde"} Jul 06 17:56:32.285597 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:32.285547 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" podStartSLOduration=3.477948483 podStartE2EDuration="22.285533594s" totalImagesPullingTime="18.807585111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:56:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:56:32.283930992 +0000 UTC m=+279.813463352" watchObservedRunningTime="2026-07-06 17:56:32.285533594 +0000 UTC m=+279.815065955" Jul 06 17:56:33.263834 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:33.263801 2587 generic.go:356] "Generic (PLEG): container finished" podID="9163efdc-c060-4603-aabf-985bf36aeb47" containerID="e895e2ffec2be666cd4ad0de64db11629bf21b48ba0e25d0f22e349f6b698bde" exitCode=0 Jul 06 17:56:33.264265 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:33.263862 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerDied","Data":"e895e2ffec2be666cd4ad0de64db11629bf21b48ba0e25d0f22e349f6b698bde"} Jul 06 17:56:34.432411 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.432385 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:34.512978 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.512953 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util\") pod \"9163efdc-c060-4603-aabf-985bf36aeb47\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " Jul 06 17:56:34.513138 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.512987 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle\") pod \"9163efdc-c060-4603-aabf-985bf36aeb47\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " Jul 06 17:56:34.513138 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.513024 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v\") pod \"9163efdc-c060-4603-aabf-985bf36aeb47\" (UID: \"9163efdc-c060-4603-aabf-985bf36aeb47\") " Jul 06 17:56:34.513633 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.513607 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle" (OuterVolumeSpecName: "bundle") pod "9163efdc-c060-4603-aabf-985bf36aeb47" (UID: "9163efdc-c060-4603-aabf-985bf36aeb47"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:56:34.515118 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.515099 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v" (OuterVolumeSpecName: "kube-api-access-nr82v") pod "9163efdc-c060-4603-aabf-985bf36aeb47" (UID: "9163efdc-c060-4603-aabf-985bf36aeb47"). InnerVolumeSpecName "kube-api-access-nr82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:56:34.518377 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.518321 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util" (OuterVolumeSpecName: "util") pod "9163efdc-c060-4603-aabf-985bf36aeb47" (UID: "9163efdc-c060-4603-aabf-985bf36aeb47"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:56:34.614254 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.614226 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:34.614254 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.614249 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9163efdc-c060-4603-aabf-985bf36aeb47-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:34.614396 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:34.614258 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr82v\" (UniqueName: \"kubernetes.io/projected/9163efdc-c060-4603-aabf-985bf36aeb47-kube-api-access-nr82v\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:35.272880 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:35.272846 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" event={"ID":"9163efdc-c060-4603-aabf-985bf36aeb47","Type":"ContainerDied","Data":"8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57"} Jul 06 17:56:35.272880 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:35.272869 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405rxtc" Jul 06 17:56:35.273085 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:35.272895 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ceed4f30db4de7e995f453fa3340796690157f5bddb94c4f35432491654dd57" Jul 06 17:56:36.930425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:36.930388 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:37.738335 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:37.738291 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:38.980773 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.980730 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j"] Jul 06 17:56:38.981253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981215 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="pull" Jul 06 17:56:38.981253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981233 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="pull" Jul 06 17:56:38.981253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981246 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="extract" Jul 06 17:56:38.981253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981253 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="extract" Jul 06 17:56:38.981462 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981269 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="util" Jul 06 17:56:38.981462 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981278 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="util" Jul 06 17:56:38.981462 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.981362 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9163efdc-c060-4603-aabf-985bf36aeb47" containerName="extract" Jul 06 17:56:38.983631 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.983612 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:38.987677 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.987659 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pgm85\"" Jul 06 17:56:38.987970 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.987955 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 17:56:38.988026 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:38.987988 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:56:39.011744 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.011696 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j"] Jul 06 17:56:39.048939 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.048909 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqvf\" (UniqueName: \"kubernetes.io/projected/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-kube-api-access-cdqvf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.049084 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.048949 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.150313 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.150286 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqvf\" (UniqueName: \"kubernetes.io/projected/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-kube-api-access-cdqvf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.150444 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.150318 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.150703 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.150686 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.162149 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.162119 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqvf\" (UniqueName: \"kubernetes.io/projected/4cecbf6c-97ec-4090-90c6-8b9adc1a24f9-kube-api-access-cdqvf\") pod \"cert-manager-operator-controller-manager-55c444bf86-r9t4j\" (UID: \"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.292847 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.292767 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" Jul 06 17:56:39.423902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:39.423880 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j"] Jul 06 17:56:39.426080 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:39.426055 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cecbf6c_97ec_4090_90c6_8b9adc1a24f9.slice/crio-c5a92dbd7b71cf4ed9ea7217b754c55b349d4d1f43d349fd021adb4df57fb425 WatchSource:0}: Error finding container c5a92dbd7b71cf4ed9ea7217b754c55b349d4d1f43d349fd021adb4df57fb425: Status 404 returned error can't find the container with id c5a92dbd7b71cf4ed9ea7217b754c55b349d4d1f43d349fd021adb4df57fb425 Jul 06 17:56:40.294486 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:40.294453 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" event={"ID":"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9","Type":"ContainerStarted","Data":"c5a92dbd7b71cf4ed9ea7217b754c55b349d4d1f43d349fd021adb4df57fb425"} Jul 06 17:56:40.800854 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:40.800803 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:40.899892 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:40.899829 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:41.301796 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:41.301765 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" event={"ID":"4cecbf6c-97ec-4090-90c6-8b9adc1a24f9","Type":"ContainerStarted","Data":"449a3b686959d6968157ee83ddcaae74ce1c0d28ccefe20a1e6176c1cdc20aab"} Jul 06 17:56:41.343771 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:41.343689 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r9t4j" podStartSLOduration=1.8113179430000002 podStartE2EDuration="3.343666351s" totalImagesPullingTime="1.532348408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:56:41.342216059 +0000 UTC m=+288.871748455" watchObservedRunningTime="2026-07-06 17:56:41.343666351 +0000 UTC m=+288.873198713" Jul 06 17:56:41.750148 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:41.750060 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:41.834995 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:41.834964 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:42.629903 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.629870 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8"] Jul 06 17:56:42.632425 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.632407 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.637892 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.637862 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 17:56:42.638011 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.637941 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 17:56:42.639191 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.638863 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m9qjl\"" Jul 06 17:56:42.651131 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.651110 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8"] Jul 06 17:56:42.679042 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.679015 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.679163 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.679056 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkn8\" (UniqueName: \"kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.679163 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.679117 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.769544 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.769505 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:42.780018 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.779993 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.780148 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.780032 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkn8\" (UniqueName: \"kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.780148 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.780070 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.780343 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.780325 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.780400 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.780383 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.789641 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.789608 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkn8\" (UniqueName: \"kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.863775 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.863742 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:42.867463 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.867437 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wcz7g"] Jul 06 17:56:42.869756 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.869741 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:42.872331 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.872315 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 17:56:42.872504 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.872493 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 17:56:42.885305 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.885253 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wcz7g"] Jul 06 17:56:42.941733 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.941694 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:42.981748 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.981695 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47p8\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-kube-api-access-x47p8\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:42.981941 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:42.981842 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.078454 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.078430 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8"] Jul 06 17:56:43.081002 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:43.080976 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda13240a_70a1_4969_8329_e4998917d16a.slice/crio-742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a WatchSource:0}: Error finding container 742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a: Status 404 returned error can't find the container with id 742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a Jul 06 17:56:43.082320 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.082300 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.082490 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.082459 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47p8\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-kube-api-access-x47p8\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.091477 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.091451 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.091563 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.091530 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47p8\" (UniqueName: \"kubernetes.io/projected/e62469ec-d54f-4aae-8ec8-f616115deefe-kube-api-access-x47p8\") pod \"cert-manager-webhook-6486ffd997-wcz7g\" (UID: \"e62469ec-d54f-4aae-8ec8-f616115deefe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.192087 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.192055 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:43.312311 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.312283 2587 generic.go:356] "Generic (PLEG): container finished" podID="da13240a-70a1-4969-8329-e4998917d16a" containerID="e44c6974e5642196f8205af10a94e17eddb55fb8faa35592713352bebe014c7e" exitCode=0 Jul 06 17:56:43.312473 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.312348 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" event={"ID":"da13240a-70a1-4969-8329-e4998917d16a","Type":"ContainerDied","Data":"e44c6974e5642196f8205af10a94e17eddb55fb8faa35592713352bebe014c7e"} Jul 06 17:56:43.312473 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.312388 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" event={"ID":"da13240a-70a1-4969-8329-e4998917d16a","Type":"ContainerStarted","Data":"742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a"} Jul 06 17:56:43.325770 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:43.325643 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wcz7g"] Jul 06 17:56:43.327686 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:43.327661 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62469ec_d54f_4aae_8ec8_f616115deefe.slice/crio-4dc561ba1941f322f687770255273a9d01fd52d1f7288a756d200a7e84400a17 WatchSource:0}: Error finding container 4dc561ba1941f322f687770255273a9d01fd52d1f7288a756d200a7e84400a17: Status 404 returned error can't find the container with id 4dc561ba1941f322f687770255273a9d01fd52d1f7288a756d200a7e84400a17 Jul 06 17:56:44.319963 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:44.319920 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" event={"ID":"e62469ec-d54f-4aae-8ec8-f616115deefe","Type":"ContainerStarted","Data":"4dc561ba1941f322f687770255273a9d01fd52d1f7288a756d200a7e84400a17"} Jul 06 17:56:44.940118 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:44.940075 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:45.127452 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:45.127417 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:46.332397 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:46.332360 2587 generic.go:356] "Generic (PLEG): container finished" podID="da13240a-70a1-4969-8329-e4998917d16a" containerID="5fc1ab3539549e64724adf40d6fba1a46ecb4787fbe3710719584bf470e9b956" exitCode=0 Jul 06 17:56:46.332845 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:46.332439 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" event={"ID":"da13240a-70a1-4969-8329-e4998917d16a","Type":"ContainerDied","Data":"5fc1ab3539549e64724adf40d6fba1a46ecb4787fbe3710719584bf470e9b956"} Jul 06 17:56:46.333991 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:46.333971 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" event={"ID":"e62469ec-d54f-4aae-8ec8-f616115deefe","Type":"ContainerStarted","Data":"21d7908bce2f50135c0ca4709f312804aef46cb13a607ecd1dd89fe2ffe37da6"} Jul 06 17:56:46.334059 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:46.334048 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:47.341309 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:47.341277 2587 generic.go:356] "Generic (PLEG): container finished" podID="da13240a-70a1-4969-8329-e4998917d16a" containerID="e6b9854c8569f5ed15d3a7df6425df66e69e2da1cec5e57230abbdc24edf8168" exitCode=0 Jul 06 17:56:47.341754 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:47.341349 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" event={"ID":"da13240a-70a1-4969-8329-e4998917d16a","Type":"ContainerDied","Data":"e6b9854c8569f5ed15d3a7df6425df66e69e2da1cec5e57230abbdc24edf8168"} Jul 06 17:56:47.361063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:47.361019 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" podStartSLOduration=2.537106168 podStartE2EDuration="5.361004543s" totalImagesPullingTime="2.823898375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:56:46.375960266 +0000 UTC m=+293.905492662" watchObservedRunningTime="2026-07-06 17:56:47.361004543 +0000 UTC m=+294.890536902" Jul 06 17:56:47.544063 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:47.544034 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:47.646904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:47.646827 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:48.471353 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.471329 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:48.531991 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.531968 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle\") pod \"da13240a-70a1-4969-8329-e4998917d16a\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " Jul 06 17:56:48.532141 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.532059 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkn8\" (UniqueName: \"kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8\") pod \"da13240a-70a1-4969-8329-e4998917d16a\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " Jul 06 17:56:48.532141 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.532101 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util\") pod \"da13240a-70a1-4969-8329-e4998917d16a\" (UID: \"da13240a-70a1-4969-8329-e4998917d16a\") " Jul 06 17:56:48.532365 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.532340 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle" (OuterVolumeSpecName: "bundle") pod "da13240a-70a1-4969-8329-e4998917d16a" (UID: "da13240a-70a1-4969-8329-e4998917d16a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:56:48.534044 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.534019 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8" (OuterVolumeSpecName: "kube-api-access-nxkn8") pod "da13240a-70a1-4969-8329-e4998917d16a" (UID: "da13240a-70a1-4969-8329-e4998917d16a"). InnerVolumeSpecName "kube-api-access-nxkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:56:48.536279 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.536250 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util" (OuterVolumeSpecName: "util") pod "da13240a-70a1-4969-8329-e4998917d16a" (UID: "da13240a-70a1-4969-8329-e4998917d16a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:56:48.632664 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.632582 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkn8\" (UniqueName: \"kubernetes.io/projected/da13240a-70a1-4969-8329-e4998917d16a-kube-api-access-nxkn8\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:48.632664 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.632610 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:48.632664 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:48.632620 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da13240a-70a1-4969-8329-e4998917d16a-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:56:49.351898 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:49.351813 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" event={"ID":"da13240a-70a1-4969-8329-e4998917d16a","Type":"ContainerDied","Data":"742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a"} Jul 06 17:56:49.351898 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:49.351867 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="742a6a7cdd9d9709e258031f9efdfb1d3800e73b82f929e78ffcb39f5acd061a" Jul 06 17:56:49.351898 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:49.351822 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdtfk8" Jul 06 17:56:50.539238 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:50.539195 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:50.650429 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:50.650397 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:52.344048 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.344018 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wcz7g" Jul 06 17:56:52.558955 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.558918 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:52.649513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.649428 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:52.906638 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.906558 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 17:56:52.906847 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.906830 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 17:56:52.917453 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:52.917427 2587 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 17:56:53.143396 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:53.143228 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:55.253387 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253348 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds"] Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253803 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="pull" Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253821 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="pull" Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253843 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="extract" Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253851 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="extract" Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253875 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="util" Jul 06 17:56:55.253888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253882 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="util" Jul 06 17:56:55.254183 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.253983 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="da13240a-70a1-4969-8329-e4998917d16a" containerName="extract" Jul 06 17:56:55.256822 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.256802 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.259600 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.259570 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 17:56:55.260550 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.260533 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 17:56:55.260636 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.260557 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jp9lg\"" Jul 06 17:56:55.265607 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.265586 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds"] Jul 06 17:56:55.387735 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.387693 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gsdk\" (UniqueName: \"kubernetes.io/projected/f63fbde7-664d-4202-86d4-cfd38098124c-kube-api-access-8gsdk\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.387909 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.387825 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f63fbde7-664d-4202-86d4-cfd38098124c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.489039 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.489003 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f63fbde7-664d-4202-86d4-cfd38098124c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.489398 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.489064 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gsdk\" (UniqueName: \"kubernetes.io/projected/f63fbde7-664d-4202-86d4-cfd38098124c-kube-api-access-8gsdk\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.489398 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.489357 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f63fbde7-664d-4202-86d4-cfd38098124c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.498035 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.498006 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gsdk\" (UniqueName: \"kubernetes.io/projected/f63fbde7-664d-4202-86d4-cfd38098124c-kube-api-access-8gsdk\") pod \"openshift-lws-operator-fd8ccff4c-f8sds\" (UID: \"f63fbde7-664d-4202-86d4-cfd38098124c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.566603 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.566574 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" Jul 06 17:56:55.694142 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.694120 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds"] Jul 06 17:56:55.696411 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:55.696383 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf63fbde7_664d_4202_86d4_cfd38098124c.slice/crio-36b43b3bf716bff2949ced6a4bee55620f7798a244430401c7bbcb0d16e60cbe WatchSource:0}: Error finding container 36b43b3bf716bff2949ced6a4bee55620f7798a244430401c7bbcb0d16e60cbe: Status 404 returned error can't find the container with id 36b43b3bf716bff2949ced6a4bee55620f7798a244430401c7bbcb0d16e60cbe Jul 06 17:56:55.698430 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.698411 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 17:56:55.754559 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.754526 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f7sf9"] Jul 06 17:56:55.758734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.758706 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:55.761373 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.761357 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jbqmb\"" Jul 06 17:56:55.769420 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.769397 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f7sf9"] Jul 06 17:56:55.894978 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.894887 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8n9z\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-kube-api-access-p8n9z\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:55.894978 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.894944 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-bound-sa-token\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:55.996134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.996087 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8n9z\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-kube-api-access-p8n9z\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:55.996134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:55.996144 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-bound-sa-token\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:56.004807 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.004778 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-bound-sa-token\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:56.005595 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.005577 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8n9z\" (UniqueName: \"kubernetes.io/projected/b895cdb4-3843-4524-bbb0-67a7728dff42-kube-api-access-p8n9z\") pod \"cert-manager-55b4d954d5-f7sf9\" (UID: \"b895cdb4-3843-4524-bbb0-67a7728dff42\") " pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:56.068559 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.068535 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f7sf9" Jul 06 17:56:56.197799 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.197765 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f7sf9"] Jul 06 17:56:56.199902 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:56:56.199866 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb895cdb4_3843_4524_bbb0_67a7728dff42.slice/crio-9ee03f2b3d9bda41db8ad0a8b4ad8dcb393ea7bc925b03d8143002a3e9e9058f WatchSource:0}: Error finding container 9ee03f2b3d9bda41db8ad0a8b4ad8dcb393ea7bc925b03d8143002a3e9e9058f: Status 404 returned error can't find the container with id 9ee03f2b3d9bda41db8ad0a8b4ad8dcb393ea7bc925b03d8143002a3e9e9058f Jul 06 17:56:56.383178 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.383132 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f7sf9" event={"ID":"b895cdb4-3843-4524-bbb0-67a7728dff42","Type":"ContainerStarted","Data":"78da14a22a8b2c023eabb86b36020d99466eeaa7b2a663ed74616321f7ebe6fe"} Jul 06 17:56:56.383178 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.383182 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f7sf9" event={"ID":"b895cdb4-3843-4524-bbb0-67a7728dff42","Type":"ContainerStarted","Data":"9ee03f2b3d9bda41db8ad0a8b4ad8dcb393ea7bc925b03d8143002a3e9e9058f"} Jul 06 17:56:56.384358 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.384332 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" event={"ID":"f63fbde7-664d-4202-86d4-cfd38098124c","Type":"ContainerStarted","Data":"36b43b3bf716bff2949ced6a4bee55620f7798a244430401c7bbcb0d16e60cbe"} Jul 06 17:56:56.401885 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:56.401791 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f7sf9" podStartSLOduration=1.401776981 podStartE2EDuration="1.401776981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:56:56.400506203 +0000 UTC m=+303.930038562" watchObservedRunningTime="2026-07-06 17:56:56.401776981 +0000 UTC m=+303.931309352" Jul 06 17:56:58.396682 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:58.396641 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" event={"ID":"f63fbde7-664d-4202-86d4-cfd38098124c","Type":"ContainerStarted","Data":"cbf62c67b43caf6dd007f9926e08b2bdb5137b793c6c396ba0c299d6dbcc7f92"} Jul 06 17:56:58.417871 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:58.417823 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f8sds" podStartSLOduration=1.326583911 podStartE2EDuration="3.417806619s" totalImagesPullingTime="2.091222708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:56:58.416017864 +0000 UTC m=+305.945550223" watchObservedRunningTime="2026-07-06 17:56:58.417806619 +0000 UTC m=+305.947338980" Jul 06 17:56:59.150372 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:59.150335 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:56:59.251138 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:56:59.251102 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:01.233255 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.233223 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd"] Jul 06 17:57:01.279964 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.279929 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd"] Jul 06 17:57:01.280122 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.280039 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.282797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.282772 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 17:57:01.283846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.283824 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 17:57:01.283944 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.283843 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m9qjl\"" Jul 06 17:57:01.344310 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.344273 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:01.445526 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.445488 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wzs\" (UniqueName: \"kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.445737 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.445595 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.445737 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.445630 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.546999 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.546909 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.546999 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.546960 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.547196 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.547003 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wzs\" (UniqueName: \"kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.547374 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.547355 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.547423 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.547388 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.556367 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.556329 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wzs\" (UniqueName: \"kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.590132 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.590096 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:01.760038 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:01.760011 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd"] Jul 06 17:57:01.761876 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:57:01.761845 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc4708_7682_45f1_9def_18886b074fef.slice/crio-f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d WatchSource:0}: Error finding container f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d: Status 404 returned error can't find the container with id f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d Jul 06 17:57:02.417892 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:02.417862 2587 generic.go:356] "Generic (PLEG): container finished" podID="effc4708-7682-45f1-9def-18886b074fef" containerID="5a8e0746ff2434d8a01b25cb0b538ec765418621f8265921d3339b6ad5b99e75" exitCode=0 Jul 06 17:57:02.418292 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:02.417953 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" event={"ID":"effc4708-7682-45f1-9def-18886b074fef","Type":"ContainerDied","Data":"5a8e0746ff2434d8a01b25cb0b538ec765418621f8265921d3339b6ad5b99e75"} Jul 06 17:57:02.418292 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:02.418005 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" event={"ID":"effc4708-7682-45f1-9def-18886b074fef","Type":"ContainerStarted","Data":"f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d"} Jul 06 17:57:04.431060 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:04.431026 2587 generic.go:356] "Generic (PLEG): container finished" podID="effc4708-7682-45f1-9def-18886b074fef" containerID="ecb28b2653205ee131a3ccf3d19a4d3caf6e078cd673b6a17e2e7bb06a062479" exitCode=0 Jul 06 17:57:04.431499 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:04.431094 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" event={"ID":"effc4708-7682-45f1-9def-18886b074fef","Type":"ContainerDied","Data":"ecb28b2653205ee131a3ccf3d19a4d3caf6e078cd673b6a17e2e7bb06a062479"} Jul 06 17:57:05.442281 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:05.442249 2587 generic.go:356] "Generic (PLEG): container finished" podID="effc4708-7682-45f1-9def-18886b074fef" containerID="d4395cde9c83476203982154a02be56f255822dcfed26d0977445e5ebda08b44" exitCode=0 Jul 06 17:57:05.442753 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:05.442334 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" event={"ID":"effc4708-7682-45f1-9def-18886b074fef","Type":"ContainerDied","Data":"d4395cde9c83476203982154a02be56f255822dcfed26d0977445e5ebda08b44"} Jul 06 17:57:06.577306 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.577278 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:06.590625 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.590600 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle\") pod \"effc4708-7682-45f1-9def-18886b074fef\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " Jul 06 17:57:06.590751 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.590678 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4wzs\" (UniqueName: \"kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs\") pod \"effc4708-7682-45f1-9def-18886b074fef\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " Jul 06 17:57:06.590751 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.590729 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util\") pod \"effc4708-7682-45f1-9def-18886b074fef\" (UID: \"effc4708-7682-45f1-9def-18886b074fef\") " Jul 06 17:57:06.591357 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.591331 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle" (OuterVolumeSpecName: "bundle") pod "effc4708-7682-45f1-9def-18886b074fef" (UID: "effc4708-7682-45f1-9def-18886b074fef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:57:06.592691 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.592661 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs" (OuterVolumeSpecName: "kube-api-access-l4wzs") pod "effc4708-7682-45f1-9def-18886b074fef" (UID: "effc4708-7682-45f1-9def-18886b074fef"). InnerVolumeSpecName "kube-api-access-l4wzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:57:06.596209 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.596177 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util" (OuterVolumeSpecName: "util") pod "effc4708-7682-45f1-9def-18886b074fef" (UID: "effc4708-7682-45f1-9def-18886b074fef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:57:06.691830 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.691794 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wzs\" (UniqueName: \"kubernetes.io/projected/effc4708-7682-45f1-9def-18886b074fef-kube-api-access-l4wzs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:57:06.691830 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.691823 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:57:06.691830 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:06.691834 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effc4708-7682-45f1-9def-18886b074fef-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:57:07.453489 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:07.453397 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" Jul 06 17:57:07.453489 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:07.453408 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnkkgd" event={"ID":"effc4708-7682-45f1-9def-18886b074fef","Type":"ContainerDied","Data":"f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d"} Jul 06 17:57:07.453489 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:07.453461 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4195b3ac5310c2fc097b1e941be504db87f2d6740eb0b18df32564fcb35e10d" Jul 06 17:57:17.941962 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:17.941922 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.079978 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.079937 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.224495 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224464 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8"] Jul 06 17:57:19.224806 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224794 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="util" Jul 06 17:57:19.224857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224808 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="util" Jul 06 17:57:19.224857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224815 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="extract" Jul 06 17:57:19.224857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224820 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="extract" Jul 06 17:57:19.224857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224831 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="pull" Jul 06 17:57:19.224857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224836 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="pull" Jul 06 17:57:19.225044 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.224929 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="effc4708-7682-45f1-9def-18886b074fef" containerName="extract" Jul 06 17:57:19.226922 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.226907 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.229580 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.229557 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 17:57:19.229767 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.229617 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 17:57:19.229918 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.229902 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 17:57:19.230003 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.229944 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2c6xq\"" Jul 06 17:57:19.230076 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.230019 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 17:57:19.243775 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.243755 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8"] Jul 06 17:57:19.284404 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.284364 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.284548 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.284451 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-manifests\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.284548 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.284508 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-tmp\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.284643 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.284616 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.284754 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.284728 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffc2q\" (UniqueName: \"kubernetes.io/projected/76f5931f-5426-4cf5-b9a8-698dec958622-kube-api-access-ffc2q\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.385612 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.385544 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.385612 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.385592 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffc2q\" (UniqueName: \"kubernetes.io/projected/76f5931f-5426-4cf5-b9a8-698dec958622-kube-api-access-ffc2q\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.385826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.385628 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.385826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.385657 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-manifests\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.385826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.385684 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-tmp\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.386094 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.386073 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-manifests\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.387928 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.387900 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76f5931f-5426-4cf5-b9a8-698dec958622-tmp\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.388143 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.388122 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.388184 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.388167 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76f5931f-5426-4cf5-b9a8-698dec958622-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.398569 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.398548 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffc2q\" (UniqueName: \"kubernetes.io/projected/76f5931f-5426-4cf5-b9a8-698dec958622-kube-api-access-ffc2q\") pod \"opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8\" (UID: \"76f5931f-5426-4cf5-b9a8-698dec958622\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.444398 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.444369 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.537632 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.537599 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:19.543680 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.543645 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.648876 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.648785 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.678167 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.678147 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8"] Jul 06 17:57:19.680418 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:57:19.680387 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f5931f_5426_4cf5_b9a8_698dec958622.slice/crio-42222614594eabb9c913d97a97edf662d592fa28310b96e4f7ecc8019d404e48 WatchSource:0}: Error finding container 42222614594eabb9c913d97a97edf662d592fa28310b96e4f7ecc8019d404e48: Status 404 returned error can't find the container with id 42222614594eabb9c913d97a97edf662d592fa28310b96e4f7ecc8019d404e48 Jul 06 17:57:19.743847 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.743813 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:19.850224 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:19.850184 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:20.513686 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:20.513633 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" event={"ID":"76f5931f-5426-4cf5-b9a8-698dec958622","Type":"ContainerStarted","Data":"42222614594eabb9c913d97a97edf662d592fa28310b96e4f7ecc8019d404e48"} Jul 06 17:57:23.530391 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:23.530356 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" event={"ID":"76f5931f-5426-4cf5-b9a8-698dec958622","Type":"ContainerStarted","Data":"e53f1d3a110c647e96f3e7e46492c60a1a508a5ffa0c5e64fc0b409ffaed67d8"} Jul 06 17:57:24.536872 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:24.536838 2587 generic.go:356] "Generic (PLEG): container finished" podID="76f5931f-5426-4cf5-b9a8-698dec958622" containerID="e53f1d3a110c647e96f3e7e46492c60a1a508a5ffa0c5e64fc0b409ffaed67d8" exitCode=0 Jul 06 17:57:24.537234 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:24.536901 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" event={"ID":"76f5931f-5426-4cf5-b9a8-698dec958622","Type":"ContainerDied","Data":"e53f1d3a110c647e96f3e7e46492c60a1a508a5ffa0c5e64fc0b409ffaed67d8"} Jul 06 17:57:25.543035 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:25.542997 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" event={"ID":"76f5931f-5426-4cf5-b9a8-698dec958622","Type":"ContainerStarted","Data":"3eecc7c25cfda572f6886ecd941c1a73589dbc3df049d92b5a944b75304b63a2"} Jul 06 17:57:25.543450 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:25.543148 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:25.580487 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:25.580437 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" podStartSLOduration=2.02747953 podStartE2EDuration="6.580418189s" totalImagesPullingTime="3.460226413s" totalInitContainerRuntime="1.092712246s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:57:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:57:25.580181627 +0000 UTC m=+333.109714018" watchObservedRunningTime="2026-07-06 17:57:25.580418189 +0000 UTC m=+333.109950549" Jul 06 17:57:25.733342 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:25.733306 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:25.832414 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:25.832379 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:31.061698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:31.061658 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:31.166651 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:31.166612 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:31.275353 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:31.275321 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:36.550306 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:36.550275 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8" Jul 06 17:57:39.419206 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:39.419167 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:39.677323 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:39.677237 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:40.259170 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:40.259140 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:40.383773 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:40.383740 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:40.482737 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:40.482687 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:41.142079 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:41.142042 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:41.269857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:41.269816 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:41.402426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:41.402339 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:42.976503 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:42.976461 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:43.104750 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.104703 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p"] Jul 06 17:57:43.112856 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.112835 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.115703 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.115677 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 17:57:43.115867 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.115685 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7wdq2\"" Jul 06 17:57:43.116824 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.116800 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 17:57:43.116931 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.116812 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 17:57:43.117078 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.117065 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 17:57:43.119045 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.119026 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p"] Jul 06 17:57:43.190338 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.190307 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tls-certs\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.190338 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.190340 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncn9\" (UniqueName: \"kubernetes.io/projected/2c8cc242-8d7f-4ba8-80df-39ab46aac003-kube-api-access-qncn9\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.190590 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.190432 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tmp\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.242966 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.242873 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:43.291643 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.291613 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tmp\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.291822 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.291678 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tls-certs\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.291822 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.291705 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncn9\" (UniqueName: \"kubernetes.io/projected/2c8cc242-8d7f-4ba8-80df-39ab46aac003-kube-api-access-qncn9\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.293911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.293880 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tmp\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.294059 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.294044 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8cc242-8d7f-4ba8-80df-39ab46aac003-tls-certs\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.300254 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.300232 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncn9\" (UniqueName: \"kubernetes.io/projected/2c8cc242-8d7f-4ba8-80df-39ab46aac003-kube-api-access-qncn9\") pod \"kube-auth-proxy-76c4c4d987-mqb8p\" (UID: \"2c8cc242-8d7f-4ba8-80df-39ab46aac003\") " pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.423654 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.423621 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" Jul 06 17:57:43.551140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.551101 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p"] Jul 06 17:57:43.554221 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:57:43.554197 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8cc242_8d7f_4ba8_80df_39ab46aac003.slice/crio-708331de0c2137cfb13e7a0aa3121ae0e8369a8ded443e176e01fee10ab09f5c WatchSource:0}: Error finding container 708331de0c2137cfb13e7a0aa3121ae0e8369a8ded443e176e01fee10ab09f5c: Status 404 returned error can't find the container with id 708331de0c2137cfb13e7a0aa3121ae0e8369a8ded443e176e01fee10ab09f5c Jul 06 17:57:43.624396 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:43.624367 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" event={"ID":"2c8cc242-8d7f-4ba8-80df-39ab46aac003","Type":"ContainerStarted","Data":"708331de0c2137cfb13e7a0aa3121ae0e8369a8ded443e176e01fee10ab09f5c"} Jul 06 17:57:44.378048 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:44.377957 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:44.466577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:44.466544 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:45.743000 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:45.742964 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:46.895774 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.895746 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn"] Jul 06 17:57:46.899374 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.899351 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:46.903240 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.903219 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xjdzk\"" Jul 06 17:57:46.903736 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.903691 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 17:57:46.904616 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.904596 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 17:57:46.904742 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.904638 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 17:57:46.912444 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:46.912420 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn"] Jul 06 17:57:47.026033 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.025993 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-manager-config\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.026213 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.026050 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.026213 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.026158 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-metrics-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.026316 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.026212 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtw5\" (UniqueName: \"kubernetes.io/projected/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-kube-api-access-rwtw5\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.126979 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.126942 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-metrics-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.127193 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.127010 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtw5\" (UniqueName: \"kubernetes.io/projected/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-kube-api-access-rwtw5\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.127193 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.127082 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-manager-config\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.127193 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.127132 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.127954 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.127925 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-manager-config\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.130140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.130110 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.130140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.130135 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-metrics-cert\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.144225 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.144192 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtw5\" (UniqueName: \"kubernetes.io/projected/b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee-kube-api-access-rwtw5\") pod \"lws-controller-manager-88fc8bffd-kdcmn\" (UID: \"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee\") " pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.211211 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.211127 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:47.380441 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.380412 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn"] Jul 06 17:57:47.381846 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:57:47.381820 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16e6f66_f5d9_4452_b3d1_c8fb87e7b7ee.slice/crio-1a55196f1839778c9d5d5996ad281edd98557e2ed90fd6ba51451456aa3c25c8 WatchSource:0}: Error finding container 1a55196f1839778c9d5d5996ad281edd98557e2ed90fd6ba51451456aa3c25c8: Status 404 returned error can't find the container with id 1a55196f1839778c9d5d5996ad281edd98557e2ed90fd6ba51451456aa3c25c8 Jul 06 17:57:47.645904 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.645864 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" event={"ID":"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee","Type":"ContainerStarted","Data":"1a55196f1839778c9d5d5996ad281edd98557e2ed90fd6ba51451456aa3c25c8"} Jul 06 17:57:47.647208 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.647180 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" event={"ID":"2c8cc242-8d7f-4ba8-80df-39ab46aac003","Type":"ContainerStarted","Data":"2fdf5c9599ff20f32a7e311650af3393b3d98feffe64665264d20313d5693f47"} Jul 06 17:57:47.667734 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:47.667669 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76c4c4d987-mqb8p" podStartSLOduration=0.973850358 podStartE2EDuration="4.667655384s" totalImagesPullingTime="3.693805026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:57:47.667116685 +0000 UTC m=+355.196649055" watchObservedRunningTime="2026-07-06 17:57:47.667655384 +0000 UTC m=+355.197187744" Jul 06 17:57:49.659350 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:49.659312 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" event={"ID":"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee","Type":"ContainerStarted","Data":"d1659e295ff958fe0577aa98493afbb51d10e2cfc855c3d94bb08b2fa547e7f8"} Jul 06 17:57:49.659857 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:49.659371 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:57:49.713286 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:49.713221 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" podStartSLOduration=2.28596948 podStartE2EDuration="3.713203556s" totalImagesPullingTime="1.427234076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:57:49.712077586 +0000 UTC m=+357.241609945" watchObservedRunningTime="2026-07-06 17:57:49.713203556 +0000 UTC m=+357.242735916" Jul 06 17:57:51.375482 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:51.375439 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:51.481365 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:51.481326 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:51.617630 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:51.617597 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:51.731225 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:51.731134 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:51.820202 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:51.820163 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:52.834603 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:52.834564 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:52.982424 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:52.982384 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:54.418315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:54.418285 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:54.513698 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:54.513659 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:54.773305 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:54.773216 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:54.860349 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:54.860314 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:57:55.046217 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:57:55.046125 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:00.666810 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:00.666781 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 17:58:08.194216 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:08.194183 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:40.882332 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:40.882289 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:40.987102 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:40.987066 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:41.155311 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:41.155216 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:41.822426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:41.822396 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:42.019826 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:42.019788 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:58:56.433535 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.433502 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6"] Jul 06 17:58:56.437087 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.437071 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.439917 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.439892 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 17:58:56.440856 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.440831 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w8b2\"" Jul 06 17:58:56.440948 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.440835 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 17:58:56.445942 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.445906 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6"] Jul 06 17:58:56.618591 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.618554 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlkn\" (UniqueName: \"kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.618825 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.618667 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.618825 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.618743 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.719928 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.719854 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlkn\" (UniqueName: \"kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.719928 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.719923 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.720114 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.719951 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.720279 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.720264 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.720334 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.720310 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.729082 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.729059 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlkn\" (UniqueName: \"kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.747058 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.747034 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:58:56.879008 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.878981 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6"] Jul 06 17:58:56.880246 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:58:56.880220 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe79d2de_9450_42e3_ac8c_18e3bd244fd4.slice/crio-39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785 WatchSource:0}: Error finding container 39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785: Status 404 returned error can't find the container with id 39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785 Jul 06 17:58:56.969571 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.969541 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerStarted","Data":"c321b1eca277650f19fd150a1e5779fc8714bc7f52ac9d85f5c358a09507f317"} Jul 06 17:58:56.969679 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:56.969597 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerStarted","Data":"39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785"} Jul 06 17:58:57.038777 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.038749 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r"] Jul 06 17:58:57.042199 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.042183 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.050932 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.050908 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r"] Jul 06 17:58:57.124509 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.124483 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.124636 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.124529 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vt9j\" (UniqueName: \"kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.124636 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.124562 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.225298 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.225263 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vt9j\" (UniqueName: \"kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.225298 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.225306 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.225505 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.225376 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.225778 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.225761 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.225823 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.225800 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.235082 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.235056 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vt9j\" (UniqueName: \"kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.366165 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.366135 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:58:57.436611 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.433203 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8"] Jul 06 17:58:57.440022 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.439999 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.445493 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.445470 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8"] Jul 06 17:58:57.506144 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.506116 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r"] Jul 06 17:58:57.507763 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:58:57.507739 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552dd939_7828_4a9d_81df_cf0af8dfe045.slice/crio-0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81 WatchSource:0}: Error finding container 0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81: Status 404 returned error can't find the container with id 0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81 Jul 06 17:58:57.527846 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.527822 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.527989 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.527856 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddslp\" (UniqueName: \"kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.527989 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.527878 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.628860 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.628786 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.628981 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.628908 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.628981 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.628929 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddslp\" (UniqueName: \"kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.629146 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.629127 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.629212 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.629194 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.638697 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.638675 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddslp\" (UniqueName: \"kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.751722 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.751681 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:58:57.856707 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.856680 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv"] Jul 06 17:58:57.860627 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.860601 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:57.879106 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.879046 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv"] Jul 06 17:58:57.889630 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.889609 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8"] Jul 06 17:58:57.890985 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:58:57.890957 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9f302c_fc63_47fb_8ced_0f3f41a97043.slice/crio-65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30 WatchSource:0}: Error finding container 65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30: Status 404 returned error can't find the container with id 65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30 Jul 06 17:58:57.931737 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.931695 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrl7x\" (UniqueName: \"kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:57.931859 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.931806 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:57.931932 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.931862 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:57.975669 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.975631 2587 generic.go:356] "Generic (PLEG): container finished" podID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerID="c321b1eca277650f19fd150a1e5779fc8714bc7f52ac9d85f5c358a09507f317" exitCode=0 Jul 06 17:58:57.975888 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.975736 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerDied","Data":"c321b1eca277650f19fd150a1e5779fc8714bc7f52ac9d85f5c358a09507f317"} Jul 06 17:58:57.977463 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.977442 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerStarted","Data":"8d4eaebf8f2ae2c85a89147bed1a29fd29f46ffedaa1c3e9c87403fe65796d77"} Jul 06 17:58:57.977575 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.977486 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerStarted","Data":"65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30"} Jul 06 17:58:57.979133 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.979104 2587 generic.go:356] "Generic (PLEG): container finished" podID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerID="16eb81ba79fcd8b920971e6893f5ce6f28b5dfeebef447b6c2002e054738a231" exitCode=0 Jul 06 17:58:57.979235 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.979150 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" event={"ID":"552dd939-7828-4a9d-81df-cf0af8dfe045","Type":"ContainerDied","Data":"16eb81ba79fcd8b920971e6893f5ce6f28b5dfeebef447b6c2002e054738a231"} Jul 06 17:58:57.979235 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:57.979189 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" event={"ID":"552dd939-7828-4a9d-81df-cf0af8dfe045","Type":"ContainerStarted","Data":"0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81"} Jul 06 17:58:58.032424 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.032390 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.032591 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.032461 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.032591 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.032573 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrl7x\" (UniqueName: \"kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.032862 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.032841 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.032862 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.032856 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.048860 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.048828 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrl7x\" (UniqueName: \"kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.170492 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.170414 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:58:58.329957 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.329934 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv"] Jul 06 17:58:58.332395 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:58:58.332368 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f981780_7213_43f9_97de_4aa3cd23d558.slice/crio-b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f WatchSource:0}: Error finding container b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f: Status 404 returned error can't find the container with id b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f Jul 06 17:58:58.986092 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.986061 2587 generic.go:356] "Generic (PLEG): container finished" podID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerID="24558b9baafbec132276687560b4dfc91aa88ce9b314932557bb91b3e5124abb" exitCode=0 Jul 06 17:58:58.986496 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.986144 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerDied","Data":"24558b9baafbec132276687560b4dfc91aa88ce9b314932557bb91b3e5124abb"} Jul 06 17:58:58.987848 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.987829 2587 generic.go:356] "Generic (PLEG): container finished" podID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerID="8d4eaebf8f2ae2c85a89147bed1a29fd29f46ffedaa1c3e9c87403fe65796d77" exitCode=0 Jul 06 17:58:58.987924 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.987905 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerDied","Data":"8d4eaebf8f2ae2c85a89147bed1a29fd29f46ffedaa1c3e9c87403fe65796d77"} Jul 06 17:58:58.989792 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.989770 2587 generic.go:356] "Generic (PLEG): container finished" podID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerID="d2f4553da51b5b50eda066922a4fe2df6de79f0ddf1be8eb15ac7e8bae70d63a" exitCode=0 Jul 06 17:58:58.989902 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.989848 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" event={"ID":"552dd939-7828-4a9d-81df-cf0af8dfe045","Type":"ContainerDied","Data":"d2f4553da51b5b50eda066922a4fe2df6de79f0ddf1be8eb15ac7e8bae70d63a"} Jul 06 17:58:58.991492 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.991474 2587 generic.go:356] "Generic (PLEG): container finished" podID="3f981780-7213-43f9-97de-4aa3cd23d558" containerID="9f48fc88e9ee3413f0dad8c71ccd96e7844ff608d95df1096ed589e6f0520fca" exitCode=0 Jul 06 17:58:58.991579 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.991507 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" event={"ID":"3f981780-7213-43f9-97de-4aa3cd23d558","Type":"ContainerDied","Data":"9f48fc88e9ee3413f0dad8c71ccd96e7844ff608d95df1096ed589e6f0520fca"} Jul 06 17:58:58.991579 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:58:58.991536 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" event={"ID":"3f981780-7213-43f9-97de-4aa3cd23d558","Type":"ContainerStarted","Data":"b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f"} Jul 06 17:59:00.001072 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.001047 2587 generic.go:356] "Generic (PLEG): container finished" podID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerID="817562bc358e571f71b1ec8bd74dcd741b32ebfa9fade6653aaba0fcc9856c9e" exitCode=0 Jul 06 17:59:00.001385 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.001109 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" event={"ID":"552dd939-7828-4a9d-81df-cf0af8dfe045","Type":"ContainerDied","Data":"817562bc358e571f71b1ec8bd74dcd741b32ebfa9fade6653aaba0fcc9856c9e"} Jul 06 17:59:00.002797 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.002775 2587 generic.go:356] "Generic (PLEG): container finished" podID="3f981780-7213-43f9-97de-4aa3cd23d558" containerID="736cf626b824b6817de9197bd19ab0348ff7d6f3fd656eff3ea809799971ed6d" exitCode=0 Jul 06 17:59:00.002896 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.002864 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" event={"ID":"3f981780-7213-43f9-97de-4aa3cd23d558","Type":"ContainerDied","Data":"736cf626b824b6817de9197bd19ab0348ff7d6f3fd656eff3ea809799971ed6d"} Jul 06 17:59:00.005338 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.005305 2587 generic.go:356] "Generic (PLEG): container finished" podID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerID="13ec9259f39589300d755d534172e8322e23ab9e82dedd986658a1ef69a56995" exitCode=0 Jul 06 17:59:00.005436 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.005353 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerDied","Data":"13ec9259f39589300d755d534172e8322e23ab9e82dedd986658a1ef69a56995"} Jul 06 17:59:00.902356 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:00.902307 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:01.012424 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.012321 2587 generic.go:356] "Generic (PLEG): container finished" podID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerID="5ed226831063cfbb9ade63538966f4900a4213e4ead1be00ed0564f651adf264" exitCode=0 Jul 06 17:59:01.012424 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.012412 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerDied","Data":"5ed226831063cfbb9ade63538966f4900a4213e4ead1be00ed0564f651adf264"} Jul 06 17:59:01.014531 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.014510 2587 generic.go:356] "Generic (PLEG): container finished" podID="3f981780-7213-43f9-97de-4aa3cd23d558" containerID="2b8b40b2049a5d04f63a73c6ab8c19492644abacce51f1af10867699efff1ec4" exitCode=0 Jul 06 17:59:01.014683 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.014663 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" event={"ID":"3f981780-7213-43f9-97de-4aa3cd23d558","Type":"ContainerDied","Data":"2b8b40b2049a5d04f63a73c6ab8c19492644abacce51f1af10867699efff1ec4"} Jul 06 17:59:01.017974 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.017949 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:01.152584 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.152557 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:59:01.167755 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.167702 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vt9j\" (UniqueName: \"kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j\") pod \"552dd939-7828-4a9d-81df-cf0af8dfe045\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " Jul 06 17:59:01.167852 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.167778 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util\") pod \"552dd939-7828-4a9d-81df-cf0af8dfe045\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " Jul 06 17:59:01.167907 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.167892 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle\") pod \"552dd939-7828-4a9d-81df-cf0af8dfe045\" (UID: \"552dd939-7828-4a9d-81df-cf0af8dfe045\") " Jul 06 17:59:01.168367 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.168342 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle" (OuterVolumeSpecName: "bundle") pod "552dd939-7828-4a9d-81df-cf0af8dfe045" (UID: "552dd939-7828-4a9d-81df-cf0af8dfe045"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:01.169646 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.169625 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j" (OuterVolumeSpecName: "kube-api-access-2vt9j") pod "552dd939-7828-4a9d-81df-cf0af8dfe045" (UID: "552dd939-7828-4a9d-81df-cf0af8dfe045"). InnerVolumeSpecName "kube-api-access-2vt9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:59:01.173388 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.173365 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util" (OuterVolumeSpecName: "util") pod "552dd939-7828-4a9d-81df-cf0af8dfe045" (UID: "552dd939-7828-4a9d-81df-cf0af8dfe045"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:01.217668 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.217643 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:59:01.269046 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.268953 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util\") pod \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " Jul 06 17:59:01.269046 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269005 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle\") pod \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " Jul 06 17:59:01.269046 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269037 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnlkn\" (UniqueName: \"kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn\") pod \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\" (UID: \"fe79d2de-9450-42e3-ac8c-18e3bd244fd4\") " Jul 06 17:59:01.269315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269258 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:01.269315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269277 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vt9j\" (UniqueName: \"kubernetes.io/projected/552dd939-7828-4a9d-81df-cf0af8dfe045-kube-api-access-2vt9j\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:01.269315 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269292 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/552dd939-7828-4a9d-81df-cf0af8dfe045-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:01.269546 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.269512 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle" (OuterVolumeSpecName: "bundle") pod "fe79d2de-9450-42e3-ac8c-18e3bd244fd4" (UID: "fe79d2de-9450-42e3-ac8c-18e3bd244fd4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:01.271119 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.271098 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn" (OuterVolumeSpecName: "kube-api-access-dnlkn") pod "fe79d2de-9450-42e3-ac8c-18e3bd244fd4" (UID: "fe79d2de-9450-42e3-ac8c-18e3bd244fd4"). InnerVolumeSpecName "kube-api-access-dnlkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:59:01.274339 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.274320 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util" (OuterVolumeSpecName: "util") pod "fe79d2de-9450-42e3-ac8c-18e3bd244fd4" (UID: "fe79d2de-9450-42e3-ac8c-18e3bd244fd4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:01.370480 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.370446 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:01.370480 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.370474 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlkn\" (UniqueName: \"kubernetes.io/projected/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-kube-api-access-dnlkn\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:01.370480 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:01.370484 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe79d2de-9450-42e3-ac8c-18e3bd244fd4-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:02.021845 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.021747 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" event={"ID":"fe79d2de-9450-42e3-ac8c-18e3bd244fd4","Type":"ContainerDied","Data":"39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785"} Jul 06 17:59:02.021845 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.021802 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39aa8d5ce63307b8e0b30181051455784faf16b4fc2b4488640eb688323a7785" Jul 06 17:59:02.021845 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.021761 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6" Jul 06 17:59:02.023939 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.023914 2587 generic.go:356] "Generic (PLEG): container finished" podID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerID="30cba23678c4b6a9b7bc12619723d54eb6b5e9ab54f1cdd3696632ce0f9f9b1a" exitCode=0 Jul 06 17:59:02.024065 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.023990 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerDied","Data":"30cba23678c4b6a9b7bc12619723d54eb6b5e9ab54f1cdd3696632ce0f9f9b1a"} Jul 06 17:59:02.025978 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.025953 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" Jul 06 17:59:02.026092 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.025992 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r" event={"ID":"552dd939-7828-4a9d-81df-cf0af8dfe045","Type":"ContainerDied","Data":"0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81"} Jul 06 17:59:02.026092 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.026033 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a36eb36762bc985d832c05e177f9193ebd0f1af506d297f8da9c41000773e81" Jul 06 17:59:02.167236 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.167209 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:59:02.278964 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.278884 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util\") pod \"3f981780-7213-43f9-97de-4aa3cd23d558\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " Jul 06 17:59:02.279133 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.278980 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrl7x\" (UniqueName: \"kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x\") pod \"3f981780-7213-43f9-97de-4aa3cd23d558\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " Jul 06 17:59:02.279133 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.278998 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle\") pod \"3f981780-7213-43f9-97de-4aa3cd23d558\" (UID: \"3f981780-7213-43f9-97de-4aa3cd23d558\") " Jul 06 17:59:02.279573 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.279550 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle" (OuterVolumeSpecName: "bundle") pod "3f981780-7213-43f9-97de-4aa3cd23d558" (UID: "3f981780-7213-43f9-97de-4aa3cd23d558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:02.280943 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.280925 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x" (OuterVolumeSpecName: "kube-api-access-zrl7x") pod "3f981780-7213-43f9-97de-4aa3cd23d558" (UID: "3f981780-7213-43f9-97de-4aa3cd23d558"). InnerVolumeSpecName "kube-api-access-zrl7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:59:02.284359 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.284322 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util" (OuterVolumeSpecName: "util") pod "3f981780-7213-43f9-97de-4aa3cd23d558" (UID: "3f981780-7213-43f9-97de-4aa3cd23d558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:02.380253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.380229 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:02.380253 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.380250 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrl7x\" (UniqueName: \"kubernetes.io/projected/3f981780-7213-43f9-97de-4aa3cd23d558-kube-api-access-zrl7x\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:02.380408 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:02.380261 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f981780-7213-43f9-97de-4aa3cd23d558-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:03.033058 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.033020 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" event={"ID":"3f981780-7213-43f9-97de-4aa3cd23d558","Type":"ContainerDied","Data":"b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f"} Jul 06 17:59:03.033058 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.033049 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv" Jul 06 17:59:03.033496 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.033081 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5744865e77c4a233b2e6bc23265fa61082d3b1bbef913e94b8bdf71d38edb4f" Jul 06 17:59:03.170202 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.170173 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:59:03.288675 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.288584 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle\") pod \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " Jul 06 17:59:03.288675 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.288623 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddslp\" (UniqueName: \"kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp\") pod \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " Jul 06 17:59:03.288933 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.288744 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util\") pod \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\" (UID: \"3c9f302c-fc63-47fb-8ced-0f3f41a97043\") " Jul 06 17:59:03.289149 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.289123 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle" (OuterVolumeSpecName: "bundle") pod "3c9f302c-fc63-47fb-8ced-0f3f41a97043" (UID: "3c9f302c-fc63-47fb-8ced-0f3f41a97043"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:03.290660 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.290637 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp" (OuterVolumeSpecName: "kube-api-access-ddslp") pod "3c9f302c-fc63-47fb-8ced-0f3f41a97043" (UID: "3c9f302c-fc63-47fb-8ced-0f3f41a97043"). InnerVolumeSpecName "kube-api-access-ddslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:59:03.293974 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.293943 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util" (OuterVolumeSpecName: "util") pod "3c9f302c-fc63-47fb-8ced-0f3f41a97043" (UID: "3c9f302c-fc63-47fb-8ced-0f3f41a97043"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:59:03.389965 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.389919 2587 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-util\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:03.389965 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.389961 2587 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9f302c-fc63-47fb-8ced-0f3f41a97043-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:03.389965 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:03.389970 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddslp\" (UniqueName: \"kubernetes.io/projected/3c9f302c-fc63-47fb-8ced-0f3f41a97043-kube-api-access-ddslp\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:04.040146 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:04.040119 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" Jul 06 17:59:04.040146 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:04.040138 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8" event={"ID":"3c9f302c-fc63-47fb-8ced-0f3f41a97043","Type":"ContainerDied","Data":"65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30"} Jul 06 17:59:04.040585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:04.040184 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fe485e48e90eda55b525e5e132196f97e13834f67dd83138fc1e4d42525c30" Jul 06 17:59:08.986535 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:08.986495 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:09.050180 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:09.050150 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:10.334511 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:10.334477 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:10.977585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:10.977547 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:11.634509 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:11.634479 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:59:11.754572 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:11.754535 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:11.857685 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:11.857638 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:11.956524 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:11.956440 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:13.717849 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:13.717814 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:18.916458 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:18.916379 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:20.362095 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:20.362061 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:21.511184 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:21.511149 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:21.767500 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:21.767408 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:22.131635 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:22.131598 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:23.555549 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:23.555510 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:26.198156 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:26.198083 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:26.302936 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:26.302902 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:27.194512 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.194480 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:27.299608 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.299577 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:27.850196 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850162 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm"] Jul 06 17:59:27.850513 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850502 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="util" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850514 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="util" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850525 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="util" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850530 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="util" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850537 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850543 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850552 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850557 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850562 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850568 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="extract" Jul 06 17:59:27.850577 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850577 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850583 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850594 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850599 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850604 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="extract" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850609 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="extract" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850620 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="util" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850625 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="util" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850630 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="util" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850635 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="util" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850641 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850646 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850653 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850658 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="pull" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850707 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="552dd939-7828-4a9d-81df-cf0af8dfe045" containerName="extract" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850729 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe79d2de-9450-42e3-ac8c-18e3bd244fd4" containerName="extract" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850735 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f981780-7213-43f9-97de-4aa3cd23d558" containerName="extract" Jul 06 17:59:27.850911 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.850742 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9f302c-fc63-47fb-8ced-0f3f41a97043" containerName="extract" Jul 06 17:59:27.851912 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.851883 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:27.855140 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.855123 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:27.858054 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.858035 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 17:59:27.858984 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.858960 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 17:59:27.859086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.859004 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 17:59:27.859086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.859013 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w8b2\"" Jul 06 17:59:27.859086 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.859021 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 17:59:27.874810 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.874783 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm"] Jul 06 17:59:27.998180 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.998146 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4z7\" (UniqueName: \"kubernetes.io/projected/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-kube-api-access-8h4z7\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:27.998346 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.998213 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:27.998346 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:27.998296 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.099567 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.099538 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4z7\" (UniqueName: \"kubernetes.io/projected/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-kube-api-access-8h4z7\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.099774 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.099601 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.099774 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.099643 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.100308 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.100252 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.102001 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.101969 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:28.102578 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.102562 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.114169 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.114147 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4z7\" (UniqueName: \"kubernetes.io/projected/bdcdd27e-0a04-4efc-96b0-b9b1e07561e1-kube-api-access-8h4z7\") pod \"kuadrant-console-plugin-66b5774985-n6smm\" (UID: \"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.164327 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.164292 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" Jul 06 17:59:28.319026 ip-10-0-129-13 kubenswrapper[2587]: W0706 17:59:28.318995 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdcdd27e_0a04_4efc_96b0_b9b1e07561e1.slice/crio-66b92852bc6f17343408b0293ad27bdadca5ac02222af7877064690dd0c61642 WatchSource:0}: Error finding container 66b92852bc6f17343408b0293ad27bdadca5ac02222af7877064690dd0c61642: Status 404 returned error can't find the container with id 66b92852bc6f17343408b0293ad27bdadca5ac02222af7877064690dd0c61642 Jul 06 17:59:28.322768 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:28.322695 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm"] Jul 06 17:59:29.165774 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:29.165736 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" event={"ID":"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1","Type":"ContainerStarted","Data":"66b92852bc6f17343408b0293ad27bdadca5ac02222af7877064690dd0c61642"} Jul 06 17:59:31.135005 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:31.134954 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:31.236544 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:31.236511 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:36.655878 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:36.655840 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c5d95fb-6rmt4" podUID="08833a90-f880-4741-9da1-81724ae1dbfe" containerName="console" containerID="cri-o://7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66" gracePeriod=15 Jul 06 17:59:36.907801 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:36.907735 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:59:37.085585 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085555 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.085783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085591 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5mns\" (UniqueName: \"kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.085783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085615 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.085783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085632 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.085783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085660 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.085783 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085748 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.086078 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.085823 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert\") pod \"08833a90-f880-4741-9da1-81724ae1dbfe\" (UID: \"08833a90-f880-4741-9da1-81724ae1dbfe\") " Jul 06 17:59:37.086134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.086092 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca" (OuterVolumeSpecName: "service-ca") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:59:37.086134 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.086110 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:59:37.086224 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.086205 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:59:37.086384 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.086362 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config" (OuterVolumeSpecName: "console-config") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:59:37.088167 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.088139 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:59:37.088352 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.088334 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns" (OuterVolumeSpecName: "kube-api-access-j5mns") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "kube-api-access-j5mns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:59:37.088460 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.088441 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "08833a90-f880-4741-9da1-81724ae1dbfe" (UID: "08833a90-f880-4741-9da1-81724ae1dbfe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187354 2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-oauth-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187385 2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08833a90-f880-4741-9da1-81724ae1dbfe-console-serving-cert\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187399 2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-console-config\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187411 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mns\" (UniqueName: \"kubernetes.io/projected/08833a90-f880-4741-9da1-81724ae1dbfe-kube-api-access-j5mns\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187421 2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-trusted-ca-bundle\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187429 2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-oauth-serving-cert\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.187431 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.187437 2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08833a90-f880-4741-9da1-81724ae1dbfe-service-ca\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 17:59:37.216006 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.215971 2587 generic.go:356] "Generic (PLEG): container finished" podID="08833a90-f880-4741-9da1-81724ae1dbfe" containerID="7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66" exitCode=0 Jul 06 17:59:37.216160 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.216030 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c5d95fb-6rmt4" Jul 06 17:59:37.216160 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.216062 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c5d95fb-6rmt4" event={"ID":"08833a90-f880-4741-9da1-81724ae1dbfe","Type":"ContainerDied","Data":"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66"} Jul 06 17:59:37.216160 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.216123 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c5d95fb-6rmt4" event={"ID":"08833a90-f880-4741-9da1-81724ae1dbfe","Type":"ContainerDied","Data":"9b32989a94c08ecb0d98490d4453f6454580e1384c311db81c7c5068683a2413"} Jul 06 17:59:37.216160 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.216152 2587 scope.go:117] "RemoveContainer" containerID="7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66" Jul 06 17:59:37.227426 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.227408 2587 scope.go:117] "RemoveContainer" containerID="7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66" Jul 06 17:59:37.227708 ip-10-0-129-13 kubenswrapper[2587]: E0706 17:59:37.227689 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66\": container with ID starting with 7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66 not found: ID does not exist" containerID="7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66" Jul 06 17:59:37.227815 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.227733 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66"} err="failed to get container status \"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66\": rpc error: code = NotFound desc = could not find container \"7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66\": container with ID starting with 7440e4a3f74d7fb1a208fc79899c8a65e1d2329e9dbb9e7db2771d8792c80d66 not found: ID does not exist" Jul 06 17:59:37.270403 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.270367 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:59:37.273665 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.273635 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c5d95fb-6rmt4"] Jul 06 17:59:37.739333 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.739291 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:37.840210 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:37.840175 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:38.817638 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:38.817603 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:38.918301 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:38.918270 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:39.077368 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:39.077272 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08833a90-f880-4741-9da1-81724ae1dbfe" path="/var/lib/kubelet/pods/08833a90-f880-4741-9da1-81724ae1dbfe/volumes" Jul 06 17:59:42.532540 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:42.532504 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:42.632656 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:42.632624 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:45.019806 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:45.019767 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:45.119533 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:45.119479 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:48.455778 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:48.455702 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:48.554557 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:48.554523 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:53.313589 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:53.313551 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" event={"ID":"bdcdd27e-0a04-4efc-96b0-b9b1e07561e1","Type":"ContainerStarted","Data":"a7976ab131f8f5044b0b73efcc9eab5ae800dbd807dd47d125907232199bcbfd"} Jul 06 17:59:53.332645 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:53.332600 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-n6smm" podStartSLOduration=2.058448889 podStartE2EDuration="26.33258495s" totalImagesPullingTime="24.274136061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:59:53.330737742 +0000 UTC m=+480.860270098" watchObservedRunningTime="2026-07-06 17:59:53.33258495 +0000 UTC m=+480.862117310" Jul 06 17:59:55.497110 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:55.497069 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 17:59:55.595684 ip-10-0-129-13 kubenswrapper[2587]: I0706 17:59:55.595646 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:00.330652 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:00.330614 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:00.427083 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:00.427046 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:02.879207 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:02.879159 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:02.979233 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:02.979197 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:06.484103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:06.484070 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:06.595883 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:06.595846 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:11.673537 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:11.673507 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:12.003467 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:12.003369 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:14.237515 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:14.237473 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:14.339787 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:14.339753 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:18.895203 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:18.895168 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:19.041996 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:19.041954 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:24.937181 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:24.937124 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:25.151165 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.151118 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:25.355574 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.355546 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:00:25.355890 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.355877 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08833a90-f880-4741-9da1-81724ae1dbfe" containerName="console" Jul 06 18:00:25.355943 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.355892 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="08833a90-f880-4741-9da1-81724ae1dbfe" containerName="console" Jul 06 18:00:25.355979 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.355964 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="08833a90-f880-4741-9da1-81724ae1dbfe" containerName="console" Jul 06 18:00:25.445775 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.445744 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:00:25.445955 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.445790 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:00:25.445955 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.445836 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.448337 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.448313 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 18:00:25.609964 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.609871 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf663702-0401-480e-8cd8-e7a58513a700-config-file\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.609964 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.609913 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvp2\" (UniqueName: \"kubernetes.io/projected/bf663702-0401-480e-8cd8-e7a58513a700-kube-api-access-wmvp2\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.710914 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.710886 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf663702-0401-480e-8cd8-e7a58513a700-config-file\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.711084 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.710928 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvp2\" (UniqueName: \"kubernetes.io/projected/bf663702-0401-480e-8cd8-e7a58513a700-kube-api-access-wmvp2\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.711594 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.711571 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf663702-0401-480e-8cd8-e7a58513a700-config-file\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.717126 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.717075 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:25.719879 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.719708 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvp2\" (UniqueName: \"kubernetes.io/projected/bf663702-0401-480e-8cd8-e7a58513a700-kube-api-access-wmvp2\") pod \"limitador-limitador-54d6d988d5-fvp29\" (UID: \"bf663702-0401-480e-8cd8-e7a58513a700\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.756677 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.756653 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:25.903023 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:25.902999 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:00:25.904760 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:00:25.904732 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf663702_0401_480e_8cd8_e7a58513a700.slice/crio-490d843400a89aae96b3954eb3e39b48ef647b1cd38bf2380bad8c53207d59c7 WatchSource:0}: Error finding container 490d843400a89aae96b3954eb3e39b48ef647b1cd38bf2380bad8c53207d59c7: Status 404 returned error can't find the container with id 490d843400a89aae96b3954eb3e39b48ef647b1cd38bf2380bad8c53207d59c7 Jul 06 18:00:26.267127 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:26.267050 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:26.489899 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:26.489862 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" event={"ID":"bf663702-0401-480e-8cd8-e7a58513a700","Type":"ContainerStarted","Data":"490d843400a89aae96b3954eb3e39b48ef647b1cd38bf2380bad8c53207d59c7"} Jul 06 18:00:29.507588 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:29.507551 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" event={"ID":"bf663702-0401-480e-8cd8-e7a58513a700","Type":"ContainerStarted","Data":"29defe6a05229a4cb59d0ae8317012da9e733da73135587048bcdcfadf4136a3"} Jul 06 18:00:29.508108 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:29.507680 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:29.527740 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:29.527657 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" podStartSLOduration=1.389047114 podStartE2EDuration="4.527638542s" totalImagesPullingTime="3.138591428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:00:29.527228122 +0000 UTC m=+517.056760482" watchObservedRunningTime="2026-07-06 18:00:29.527638542 +0000 UTC m=+517.057170927" Jul 06 18:00:35.618385 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:35.618352 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:40.513163 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:40.513132 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fvp29" Jul 06 18:00:44.056655 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:44.056624 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:44.157533 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:44.157501 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:45.887788 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:45.887754 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:57.782405 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:57.782372 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:00:57.891774 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:00:57.891740 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:05.559857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:05.559822 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:05.678785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:05.678754 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:08.977164 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:08.977113 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:09.195273 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:09.195238 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:40.421049 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:40.420999 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:41.165706 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:41.165659 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:41.450150 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:41.450049 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:41.546075 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:41.546039 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:01:41.572393 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:41.572356 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:50.744785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:50.744741 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:51.417418 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:51.417385 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:52.940919 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:52.940888 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:01:52.941913 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:52.941888 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:01:55.285120 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:55.285091 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:57.496393 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:57.496354 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:01:57.997038 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:01:57.997005 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:02:09.772990 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:09.772942 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:09.845754 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:09.845701 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:12.635847 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:12.635815 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:13.488941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:13.488906 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:02:21.226155 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:21.226073 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:23.143876 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:23.143844 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:24.195000 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:24.194964 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:02:34.896200 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.896165 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl"] Jul 06 18:02:34.899477 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.899459 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.903349 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.903328 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l6szp\"" Jul 06 18:02:34.903349 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.903342 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 18:02:34.903555 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.903330 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 18:02:34.903555 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.903417 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 18:02:34.910870 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.910843 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.910974 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.910892 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.910974 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.910948 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68g8\" (UniqueName: \"kubernetes.io/projected/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-kube-api-access-m68g8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.911058 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.911044 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.911098 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.911072 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.911135 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.911107 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:34.911638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.911610 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl"] Jul 06 18:02:34.945926 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:34.945896 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:35.012156 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012119 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68g8\" (UniqueName: \"kubernetes.io/projected/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-kube-api-access-m68g8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012318 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012180 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012318 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012269 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012318 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012293 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012433 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012321 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012499 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012475 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012553 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012505 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012629 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012609 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.012844 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.012822 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.014534 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.014516 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.014823 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.014805 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.020317 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.020294 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68g8\" (UniqueName: \"kubernetes.io/projected/7a26f793-5ea5-4ca9-ad40-9d6dd02eae77-kube-api-access-m68g8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl\" (UID: \"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.210823 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.210734 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:35.347693 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.347668 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl"] Jul 06 18:02:35.349191 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:02:35.349161 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a26f793_5ea5_4ca9_ad40_9d6dd02eae77.slice/crio-a1bc53ff00f5fcb81527590fa59e38706b4c31036cafc00169d1c471f1c86e5d WatchSource:0}: Error finding container a1bc53ff00f5fcb81527590fa59e38706b4c31036cafc00169d1c471f1c86e5d: Status 404 returned error can't find the container with id a1bc53ff00f5fcb81527590fa59e38706b4c31036cafc00169d1c471f1c86e5d Jul 06 18:02:35.350418 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:35.350403 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 18:02:36.139881 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:36.139845 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" event={"ID":"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77","Type":"ContainerStarted","Data":"a1bc53ff00f5fcb81527590fa59e38706b4c31036cafc00169d1c471f1c86e5d"} Jul 06 18:02:36.299427 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:36.299391 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:02:37.146638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:37.146608 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" event={"ID":"7a26f793-5ea5-4ca9-ad40-9d6dd02eae77","Type":"ContainerStarted","Data":"8128f2a6f85cffc089812be234966d03d603bfb00d4b52f1592ef7ba7eeb997b"} Jul 06 18:02:37.169651 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:37.169612 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" podStartSLOduration=2.102591775 podStartE2EDuration="3.169596463s" totalImagesPullingTime="1.067004688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:02:37.168074606 +0000 UTC m=+644.697606968" watchObservedRunningTime="2026-07-06 18:02:37.169596463 +0000 UTC m=+644.699128832" Jul 06 18:02:45.211857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:45.211819 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:45.212380 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:45.211880 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:45.224401 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:45.224378 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:46.200961 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:46.200929 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl" Jul 06 18:02:53.168107 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:53.168047 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:02:55.203214 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:55.203183 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:02:59.673930 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:02:59.673895 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:03:01.885921 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:03:01.885890 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:03:13.116401 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:03:13.116360 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:03:25.341289 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:03:25.341253 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:18.661943 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:18.661909 2587 generic.go:356] "Generic (PLEG): container finished" podID="b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee" containerID="d1659e295ff958fe0577aa98493afbb51d10e2cfc855c3d94bb08b2fa547e7f8" exitCode=1 Jul 06 18:04:18.662460 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:18.662003 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" event={"ID":"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee","Type":"ContainerDied","Data":"d1659e295ff958fe0577aa98493afbb51d10e2cfc855c3d94bb08b2fa547e7f8"} Jul 06 18:04:18.662460 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:18.662433 2587 scope.go:117] "RemoveContainer" containerID="d1659e295ff958fe0577aa98493afbb51d10e2cfc855c3d94bb08b2fa547e7f8" Jul 06 18:04:19.669424 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:19.669383 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" event={"ID":"b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee","Type":"ContainerStarted","Data":"5ef661aa4129a798cc0f78591c1beda59a721b73bae4af3389aeced5122bdd43"} Jul 06 18:04:19.669835 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:19.669586 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 18:04:25.782866 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:25.782807 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:30.677419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:30.677384 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-88fc8bffd-kdcmn" Jul 06 18:04:32.807267 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:32.807222 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:32.968641 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:32.968604 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:33.959792 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:33.959760 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:34.097103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:34.097067 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:52.002141 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:52.002101 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:52.142229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:52.142184 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:53.006663 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:53.006623 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:53.983568 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:53.983530 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:04:54.083610 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:04:54.083573 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:05:00.508157 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:00.508121 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:11.992306 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:11.992270 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:19.492407 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:19.492323 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:29.888823 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:29.888790 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:38.756003 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:38.755968 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:05:38.855258 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:38.855221 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:05:41.815821 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:41.815790 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:48.099021 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:48.098985 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:05:48.355005 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:48.354924 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:05:49.550564 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:05:49.550521 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:11.196344 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:11.196313 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:06:24.492932 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:24.492898 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:06:30.414271 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:30.414237 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:30.514668 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:30.514634 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:30.657542 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:30.657499 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:37.824465 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:37.824432 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:37.943870 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:37.943833 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:38.107057 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:38.106961 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:38.361968 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:38.361869 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:44.230423 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:44.230334 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:44.345783 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:44.345760 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:06:52.975324 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:52.975293 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:06:52.979105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:06:52.979079 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:07:00.799692 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:00.799664 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:07:10.497679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:10.497632 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:07:19.488625 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:19.488592 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:07:30.488839 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:30.488806 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:07:39.894934 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:39.894898 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:07:51.308247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:07:51.308213 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:08:01.798420 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:08:01.798382 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:08:10.996181 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:08:10.996144 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:08:29.353814 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:08:29.353727 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:08:33.331394 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:08:33.331358 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:08:45.087538 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:08:45.087502 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:09:09.393067 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:09.393034 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:09:19.299415 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:19.299380 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:09:34.597576 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:34.597544 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:09:46.001190 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:46.001109 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:09:52.984105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:52.984060 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:09:56.120037 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:09:56.120001 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:10:05.605260 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:10:05.605229 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:10:14.593464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:10:14.593429 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:10:58.631666 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:10:58.631624 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:10:58.732193 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:10:58.732162 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:11:41.026458 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:11:41.026365 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:11:41.085295 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:11:41.085265 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:11:53.010587 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:11:53.010557 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:11:53.014758 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:11:53.014738 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:12:13.404638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:12:13.404603 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:12:22.190605 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:12:22.190572 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:12:34.907009 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:12:34.904922 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:12:47.098632 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:12:47.098554 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:12:58.035407 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:12:58.035374 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:07.301410 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:07.301377 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:14.887726 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:14.887690 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:23.810377 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:23.810341 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:32.593182 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:32.593148 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:43.193561 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:43.193532 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:50.396837 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:50.396803 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:13:59.488198 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:13:59.488161 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:14:23.088053 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:14:23.087973 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:14:32.490727 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:14:32.490681 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:14:40.134037 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:14:40.133999 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:14:42.089853 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:14:42.089819 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:14:54.291636 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:14:54.291599 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:00.143748 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.143699 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722695-6ttqz"] Jul 06 18:15:00.147147 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.147130 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:00.150920 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.150901 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-wkkbb\"" Jul 06 18:15:00.165052 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.165027 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722695-6ttqz"] Jul 06 18:15:00.207341 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.207307 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpxn\" (UniqueName: \"kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn\") pod \"maas-api-key-cleanup-29722695-6ttqz\" (UID: \"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb\") " pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:00.308522 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.308487 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpxn\" (UniqueName: \"kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn\") pod \"maas-api-key-cleanup-29722695-6ttqz\" (UID: \"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb\") " pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:00.320545 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.320514 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpxn\" (UniqueName: \"kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn\") pod \"maas-api-key-cleanup-29722695-6ttqz\" (UID: \"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb\") " pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:00.460251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.460145 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:00.592356 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.592327 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722695-6ttqz"] Jul 06 18:15:00.593890 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:15:00.593858 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a105b8_8714_4e19_9d0f_7f15f6fa5ddb.slice/crio-5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f WatchSource:0}: Error finding container 5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f: Status 404 returned error can't find the container with id 5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f Jul 06 18:15:00.595977 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.595958 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 18:15:00.995875 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:00.995839 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" event={"ID":"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb","Type":"ContainerStarted","Data":"5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f"} Jul 06 18:15:03.194769 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:03.194736 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:04.017284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:04.017243 2587 generic.go:356] "Generic (PLEG): container finished" podID="37a105b8-8714-4e19-9d0f-7f15f6fa5ddb" containerID="4dea8fcddc2df403685c0ca5d69eaef5267f1b8d12b2750deccf58b9ef241eb6" exitCode=0 Jul 06 18:15:04.017458 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:04.017329 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" event={"ID":"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb","Type":"ContainerDied","Data":"4dea8fcddc2df403685c0ca5d69eaef5267f1b8d12b2750deccf58b9ef241eb6"} Jul 06 18:15:05.149938 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:05.149917 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:05.254307 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:05.254277 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbpxn\" (UniqueName: \"kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn\") pod \"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb\" (UID: \"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb\") " Jul 06 18:15:05.256501 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:05.256473 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn" (OuterVolumeSpecName: "kube-api-access-pbpxn") pod "37a105b8-8714-4e19-9d0f-7f15f6fa5ddb" (UID: "37a105b8-8714-4e19-9d0f-7f15f6fa5ddb"). InnerVolumeSpecName "kube-api-access-pbpxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:15:05.355327 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:05.355275 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpxn\" (UniqueName: \"kubernetes.io/projected/37a105b8-8714-4e19-9d0f-7f15f6fa5ddb-kube-api-access-pbpxn\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:15:06.029027 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:06.029000 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" Jul 06 18:15:06.029203 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:06.028997 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722695-6ttqz" event={"ID":"37a105b8-8714-4e19-9d0f-7f15f6fa5ddb","Type":"ContainerDied","Data":"5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f"} Jul 06 18:15:06.029203 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:06.029122 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5955d82159ad4b34659d8c4239223954de82516ba1ff11110e50b362d1c5110f" Jul 06 18:15:10.097068 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:10.097036 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:15:10.995135 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:10.995100 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:13.771467 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:13.771430 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:15:20.097089 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:20.097059 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:29.089687 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:29.089652 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:37.490235 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:37.490202 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:47.614296 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:47.614220 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:15:57.009473 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:15:57.009438 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:10.691885 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:10.691853 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:19.892105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:19.892070 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:29.302441 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:29.302404 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:39.898260 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:39.898225 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:50.490524 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:50.490488 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:16:53.055104 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:53.055075 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:16:53.061757 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:16:53.061735 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:17:06.088814 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:06.088780 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:17:14.502253 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:14.502178 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:17:23.996526 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:23.996490 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:17:39.404647 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:39.404611 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:17:50.088370 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:50.088322 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:17:54.281013 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:54.280962 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:17:58.887978 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:17:58.887944 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:03.174729 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:03.174649 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:11.464008 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:11.463976 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:12.185727 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:12.185670 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:16.604136 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:16.604100 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:19.347267 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:19.347230 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:19.574881 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:19.574851 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:19.674757 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:19.674650 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:20.027047 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:20.026966 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:20.052358 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:20.052328 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:26.118576 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:26.118541 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:26.219637 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:26.219612 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:26.324851 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:26.324820 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:26.422305 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:26.422226 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:27.431507 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:27.431475 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:27.521666 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:27.521639 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:27.593110 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:27.593077 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:27.941329 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:27.941293 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:28.042971 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:28.042937 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:28.145657 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:28.145623 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:30.129766 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:30.129698 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:30.995902 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:30.995861 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:32.816372 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:32.816345 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:32.968347 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:32.968312 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:33.080864 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:33.080770 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:33.169401 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:33.169368 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:34.410988 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:34.410935 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:39.465007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:39.464969 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:39.530085 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:39.530058 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:39.703690 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:39.703663 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:39.803811 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:39.803786 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:40.839414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:40.839383 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:40.909886 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:40.909857 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:40.970172 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:40.970143 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:41.071272 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:41.071240 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:41.380509 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:41.380473 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:41.480420 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:41.480387 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:42.896908 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:42.896877 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:43.196858 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:43.196799 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:46.003103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:46.003021 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:46.382923 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:46.382890 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:52.135471 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:52.135436 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:52.240437 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:52.240390 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:53.864982 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:53.864945 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:54.395230 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:54.395191 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:18:58.301397 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:58.301358 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:58.391900 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:58.391872 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:58.526552 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:58.526521 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:58.623639 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:58.623531 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:59.630260 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:59.630228 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:59.733928 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:59.733901 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:59.845433 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:59.845402 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:18:59.958249 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:18:59.958167 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:00.309785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:00.309759 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:00.410042 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:00.410010 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:01.491819 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:01.491783 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:19:01.761394 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:01.761359 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:04.099154 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:04.099123 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:04.288660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:04.288628 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:04.390891 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:04.390805 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:04.521009 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:04.520978 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:06.292633 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:06.292603 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:10.225418 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:10.225380 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:10.279419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:10.279387 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:10.590382 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:10.590348 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:10.686067 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:10.686030 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:11.060500 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:11.060470 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:11.160189 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:11.160156 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:11.256438 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:11.256409 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:11.891182 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:11.891151 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:19:12.754100 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:12.754069 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:19.015544 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:19.015502 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:19.122559 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:19.122527 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:19.223241 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:19.223201 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:19.318807 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:19.318765 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:19.743627 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:19.743547 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:45.791384 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:45.791347 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:51.799419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:51.799377 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:51.901457 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:51.901429 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:19:51.999545 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:19:51.999510 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:07.939362 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:07.939329 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:13.863597 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:13.863565 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:13.916886 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:13.916860 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:14.020439 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:14.020403 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:14.132865 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:14.132780 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:24.620061 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:24.620030 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:25.468820 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:25.468785 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:31.683887 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:31.683851 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:31.794134 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:31.794099 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:31.884177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:31.884145 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:31.984804 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:31.984704 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:32.799327 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:32.799296 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:32.895866 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:32.895835 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:32.996277 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:32.996242 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:34.799205 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:34.799170 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:35.291737 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:35.291689 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:20:40.998114 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:40.998077 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:41.057638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:41.057610 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:41.705777 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:41.705742 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:44.291443 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:44.291410 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:20:47.777842 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:47.777805 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:47.838791 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:47.838763 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:48.018595 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:48.018558 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:48.345152 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:48.345120 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:48.687388 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:48.687303 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:51.588913 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:51.588882 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:51.665497 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:51.665469 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:51.738740 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:51.738686 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:51.868357 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:51.868267 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:52.146053 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:52.145967 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:52.247404 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:52.247377 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:56.113545 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:56.113511 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:56.786229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:56.786190 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:56.881755 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:56.881682 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:57.212087 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:57.212060 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:20:57.578977 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:20:57.578943 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:21:03.652197 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:03.652159 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:03.754088 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:03.754051 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:04.220672 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:04.220645 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:04.580595 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:04.580561 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:04.638637 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:04.638604 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:07.995242 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:07.995205 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:21:10.202567 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:10.202532 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:10.300160 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:10.300130 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.110621 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.110594 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.172774 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.172742 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.370514 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.370409 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.590663 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.590627 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.707084 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.707013 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.800375 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.800346 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:11.941482 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:11.941442 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:13.589427 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:13.589398 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:13.689371 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:13.689345 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:18.087747 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:18.087699 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:18.193375 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:18.193342 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:18.292280 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:18.292251 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:18.459754 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:18.459656 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:24.561253 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:24.561218 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:24.661248 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:24.661219 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:31.405964 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:31.405927 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:31.810397 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:31.810366 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:32.221897 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:32.221799 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:34.771005 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:34.770948 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:34.872590 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:34.872543 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:34.980809 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:34.980762 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:35.235587 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:35.235555 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:35.528700 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:35.528595 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:35.629274 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:35.629229 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:39.318915 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:39.318883 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:39.746665 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:39.746584 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:21:53.091520 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:53.091497 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:21:53.108108 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:53.102999 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:21:55.994728 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:55.994682 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:21:55.995120 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:55.995034 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a105b8-8714-4e19-9d0f-7f15f6fa5ddb" containerName="cleanup" Jul 06 18:21:55.995120 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:55.995045 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a105b8-8714-4e19-9d0f-7f15f6fa5ddb" containerName="cleanup" Jul 06 18:21:55.995120 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:55.995111 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a105b8-8714-4e19-9d0f-7f15f6fa5ddb" containerName="cleanup" Jul 06 18:21:55.997890 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:55.997875 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.000584 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.000558 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12711072\"/\"openshift-service-ca.crt\"" Jul 06 18:21:56.000783 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.000558 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12711072\"/\"kube-root-ca.crt\"" Jul 06 18:21:56.001569 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.001553 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12711072\"/\"default-dockercfg-rsts9\"" Jul 06 18:21:56.001634 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.001617 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12711072\"/\"e2e-lifecycle-model-12711072-kserve-self-signed-certs\"" Jul 06 18:21:56.007679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.007655 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:21:56.015528 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.015500 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:21:56.164420 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164383 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.164420 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164419 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.164651 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164446 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.164651 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164508 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.164651 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164553 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.164651 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.164590 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnlh\" (UniqueName: \"kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266011 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266045 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266103 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266073 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266384 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266207 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266384 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266252 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266384 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266295 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnlh\" (UniqueName: \"kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266533 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266508 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266591 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266552 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.266645 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.266599 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.268386 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.268356 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.268535 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.268518 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.274854 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.274829 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnlh\" (UniqueName: \"kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh\") pod \"e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.307924 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.307886 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:21:56.440058 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:56.440031 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:21:56.441241 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:21:56.441212 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cee426_e120_4580_9aa7_9a4192eb962f.slice/crio-a43a3b53d937821ef84b1cc31c0f480722184a0117da1ff12d16468c816b7a71 WatchSource:0}: Error finding container a43a3b53d937821ef84b1cc31c0f480722184a0117da1ff12d16468c816b7a71: Status 404 returned error can't find the container with id a43a3b53d937821ef84b1cc31c0f480722184a0117da1ff12d16468c816b7a71 Jul 06 18:21:57.143551 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:57.143516 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" event={"ID":"55cee426-e120-4580-9aa7-9a4192eb962f","Type":"ContainerStarted","Data":"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038"} Jul 06 18:21:57.143551 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:57.143558 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" event={"ID":"55cee426-e120-4580-9aa7-9a4192eb962f","Type":"ContainerStarted","Data":"a43a3b53d937821ef84b1cc31c0f480722184a0117da1ff12d16468c816b7a71"} Jul 06 18:21:57.162990 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:57.162937 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" podStartSLOduration=2.162920917 podStartE2EDuration="2.162920917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:21:57.162087199 +0000 UTC m=+1804.691619558" watchObservedRunningTime="2026-07-06 18:21:57.162920917 +0000 UTC m=+1804.692453302" Jul 06 18:21:59.086930 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:21:59.086896 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:22:06.308967 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:06.308938 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:06.315210 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:06.308987 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:06.321636 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:06.321613 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:07.207310 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:07.207281 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:13.798262 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:13.798223 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:22:16.936427 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:16.936387 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:22:17.303079 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:17.303051 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:22:23.031969 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:23.031937 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:22:23.898149 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:23.898116 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:22:27.994644 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:27.994605 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:22:28.041631 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:28.041600 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:22:32.825140 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:32.825101 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:22:43.015495 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.015450 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" podUID="55cee426-e120-4580-9aa7-9a4192eb962f" containerName="main" containerID="cri-o://94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038" gracePeriod=45 Jul 06 18:22:43.261238 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.261213 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:43.361369 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361337 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361415 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361451 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361510 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361543 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mnlh\" (UniqueName: \"kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361596 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs\") pod \"55cee426-e120-4580-9aa7-9a4192eb962f\" (UID: \"55cee426-e120-4580-9aa7-9a4192eb962f\") " Jul 06 18:22:43.361802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361632 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache" (OuterVolumeSpecName: "model-cache") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:22:43.361802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361649 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:22:43.361941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361823 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home" (OuterVolumeSpecName: "home") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:22:43.361941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361921 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:43.361941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361934 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:43.361941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.361943 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:43.363801 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.363774 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm" (OuterVolumeSpecName: "dshm") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:22:43.363875 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.363838 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh" (OuterVolumeSpecName: "kube-api-access-2mnlh") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "kube-api-access-2mnlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:22:43.363914 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.363896 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55cee426-e120-4580-9aa7-9a4192eb962f" (UID: "55cee426-e120-4580-9aa7-9a4192eb962f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:22:43.387791 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.387761 2587 generic.go:356] "Generic (PLEG): container finished" podID="55cee426-e120-4580-9aa7-9a4192eb962f" containerID="94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038" exitCode=0 Jul 06 18:22:43.387911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.387793 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" event={"ID":"55cee426-e120-4580-9aa7-9a4192eb962f","Type":"ContainerDied","Data":"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038"} Jul 06 18:22:43.387911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.387809 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" Jul 06 18:22:43.387911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.387819 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc" event={"ID":"55cee426-e120-4580-9aa7-9a4192eb962f","Type":"ContainerDied","Data":"a43a3b53d937821ef84b1cc31c0f480722184a0117da1ff12d16468c816b7a71"} Jul 06 18:22:43.387911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.387836 2587 scope.go:117] "RemoveContainer" containerID="94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038" Jul 06 18:22:43.396459 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.396444 2587 scope.go:117] "RemoveContainer" containerID="94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038" Jul 06 18:22:43.396659 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:22:43.396642 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038\": container with ID starting with 94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038 not found: ID does not exist" containerID="94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038" Jul 06 18:22:43.396733 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.396666 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038"} err="failed to get container status \"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038\": rpc error: code = NotFound desc = could not find container \"94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038\": container with ID starting with 94b21e156500554415e9ff123b84b460a6be4e838cd2d50335da59d8b57b7038 not found: ID does not exist" Jul 06 18:22:43.408722 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.408691 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:22:43.414579 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.414560 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12711072/e2e-lifecycle-model-12711072-kserve-d9df8bf5-b67wc"] Jul 06 18:22:43.463027 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.463002 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cee426-e120-4580-9aa7-9a4192eb962f-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:43.463027 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.463024 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mnlh\" (UniqueName: \"kubernetes.io/projected/55cee426-e120-4580-9aa7-9a4192eb962f-kube-api-access-2mnlh\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:43.463170 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:43.463035 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cee426-e120-4580-9aa7-9a4192eb962f-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:22:45.076568 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:22:45.076534 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cee426-e120-4580-9aa7-9a4192eb962f" path="/var/lib/kubelet/pods/55cee426-e120-4580-9aa7-9a4192eb962f/volumes" Jul 06 18:23:04.200137 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:04.200101 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:05.521095 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:05.521065 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:06.295111 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:06.295070 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:23:11.473589 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:11.473559 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:11.531877 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:11.531842 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:11.762246 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:11.762165 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:12.003450 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:12.003417 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:13.107934 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:13.107850 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:13.204881 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:13.204848 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:13.307686 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:13.307647 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:14.039268 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:14.039235 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:14.138782 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:14.138748 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:14.237920 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:14.237847 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:15.624834 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:15.624789 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:15.700847 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:15.700816 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:23:21.984461 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:21.984422 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:22.084368 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:22.084336 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:22.700938 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:22.700904 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:27.194060 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:27.194025 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:23:28.961973 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:28.961939 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:29.021287 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:29.021258 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:29.089453 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:29.089422 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:29.188880 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:29.188849 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:39.675926 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:39.675860 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:40.020081 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:40.019994 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:42.464947 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:42.464910 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:23:42.566638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:23:42.566609 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:01.053229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:01.053189 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:07.226163 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:07.226132 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:07.280995 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:07.280962 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:07.663172 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:07.663145 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:08.051989 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:08.051959 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:08.146702 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:08.146672 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:09.674295 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:09.674262 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:10.523046 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:10.523012 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:10.637566 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:10.637533 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:10.820801 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:10.820771 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:11.188974 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:11.188882 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:11.326968 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:11.326931 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:11.425894 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:11.425866 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:14.893137 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:14.893106 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:24:15.055546 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:15.055513 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:15.396337 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:15.396307 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:15.730825 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:15.730743 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:16.224762 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:16.224702 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:18.581359 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:18.581328 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:18.663323 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:18.663288 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:18.730348 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:18.730322 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:18.855252 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:18.855169 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:19.261331 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:19.261232 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:19.358330 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:19.358297 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:23.116359 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:23.116330 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:25.502673 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:25.502640 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:24:32.201789 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:32.201750 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:32.302148 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:32.302114 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:35.927268 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.927234 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:24:35.927639 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.927583 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cee426-e120-4580-9aa7-9a4192eb962f" containerName="main" Jul 06 18:24:35.927639 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.927594 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cee426-e120-4580-9aa7-9a4192eb962f" containerName="main" Jul 06 18:24:35.927727 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.927651 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cee426-e120-4580-9aa7-9a4192eb962f" containerName="main" Jul 06 18:24:35.930606 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.930591 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:35.933501 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.933480 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9cadd\"/\"kube-root-ca.crt\"" Jul 06 18:24:35.934435 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.934411 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9cadd\"/\"default-dockercfg-fqbvw\"" Jul 06 18:24:35.934543 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.934491 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9cadd\"/\"openshift-service-ca.crt\"" Jul 06 18:24:35.934737 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.934704 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9cadd\"/\"test-model-a9cadd-kserve-self-signed-certs\"" Jul 06 18:24:35.941044 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:35.941025 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:24:36.014558 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014526 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.014746 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014587 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvwv\" (UniqueName: \"kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.014746 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014611 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.014746 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014643 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.014746 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014685 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.014940 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.014772 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.036212 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.036185 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:36.115659 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115627 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvwv\" (UniqueName: \"kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.115842 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115663 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.115842 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115701 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.115842 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115757 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.115842 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115782 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.116063 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.115916 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.116173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.116152 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.116313 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.116204 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.116313 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.116216 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.117848 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.117831 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.118191 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.118175 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.125965 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.125941 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvwv\" (UniqueName: \"kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv\") pod \"test-model-a9cadd-kserve-845ddfcd49-qw8h8\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.240290 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.240214 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:36.371755 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.371704 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:24:36.372806 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:24:36.372781 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aeb2e01_1794_43b9_b3fa_e93c29187964.slice/crio-dfec70c14566445dd05f98dd061d001f262e008147d72b32fe045ecb851c2968 WatchSource:0}: Error finding container dfec70c14566445dd05f98dd061d001f262e008147d72b32fe045ecb851c2968: Status 404 returned error can't find the container with id dfec70c14566445dd05f98dd061d001f262e008147d72b32fe045ecb851c2968 Jul 06 18:24:36.959785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.959748 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" event={"ID":"1aeb2e01-1794-43b9-b3fa-e93c29187964","Type":"ContainerStarted","Data":"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a"} Jul 06 18:24:36.960245 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.959797 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" event={"ID":"1aeb2e01-1794-43b9-b3fa-e93c29187964","Type":"ContainerStarted","Data":"dfec70c14566445dd05f98dd061d001f262e008147d72b32fe045ecb851c2968"} Jul 06 18:24:36.981315 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:36.981273 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" podStartSLOduration=1.981260284 podStartE2EDuration="1.981260284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:24:36.980072102 +0000 UTC m=+1964.509604462" watchObservedRunningTime="2026-07-06 18:24:36.981260284 +0000 UTC m=+1964.510792622" Jul 06 18:24:40.972990 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:40.972954 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:41.072626 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:41.072592 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:41.904519 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:41.904486 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:24:46.240660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:46.240580 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:46.240660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:46.240625 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:46.253226 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:46.253198 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:47.022540 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:47.022507 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:24:57.806852 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.806821 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:24:57.810108 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.810092 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.813533 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.813501 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd9691\"/\"kube-root-ca.crt\"" Jul 06 18:24:57.813678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.813640 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd9691\"/\"openshift-service-ca.crt\"" Jul 06 18:24:57.814633 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.814612 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd9691\"/\"default-dockercfg-mrrnw\"" Jul 06 18:24:57.814633 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.814628 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd9691\"/\"test-model-cd9691-kserve-self-signed-certs\"" Jul 06 18:24:57.821938 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.821920 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:24:57.842876 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.842851 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:24:57.911145 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911119 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.911251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911157 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.911251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911173 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.911328 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911261 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.911328 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911295 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:57.911328 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:57.911315 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012264 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012233 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012275 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012299 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012349 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012371 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012414 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012394 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012700 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012677 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012798 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012706 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.012798 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.012779 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.014504 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.014485 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.014804 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.014787 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.021419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.021396 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47\") pod \"test-model-cd9691-kserve-6bb9b96f45-m6f7l\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.119885 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.119802 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:24:58.253693 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:58.253659 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:24:58.255208 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:24:58.255177 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod059edaf5_4284_4f51_8f3a_0b21832c6592.slice/crio-e6e18fc359f99d12e072a1d3e036f8d62dad69614eb36766f024dd06ab3a7a2c WatchSource:0}: Error finding container e6e18fc359f99d12e072a1d3e036f8d62dad69614eb36766f024dd06ab3a7a2c: Status 404 returned error can't find the container with id e6e18fc359f99d12e072a1d3e036f8d62dad69614eb36766f024dd06ab3a7a2c Jul 06 18:24:59.079522 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:59.079490 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" event={"ID":"059edaf5-4284-4f51-8f3a-0b21832c6592","Type":"ContainerStarted","Data":"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825"} Jul 06 18:24:59.079910 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:59.079529 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" event={"ID":"059edaf5-4284-4f51-8f3a-0b21832c6592","Type":"ContainerStarted","Data":"e6e18fc359f99d12e072a1d3e036f8d62dad69614eb36766f024dd06ab3a7a2c"} Jul 06 18:24:59.102131 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:24:59.102083 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" podStartSLOduration=2.102070179 podStartE2EDuration="2.102070179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:24:59.101253137 +0000 UTC m=+1986.630785496" watchObservedRunningTime="2026-07-06 18:24:59.102070179 +0000 UTC m=+1986.631602539" Jul 06 18:25:02.094258 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:02.094225 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:25:08.119970 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:08.119933 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:08.120518 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:08.119983 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:08.132670 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:08.132647 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:09.140795 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:09.140768 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:15.704099 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:15.704063 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:25:16.609400 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:16.609366 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:25:16.625993 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:16.625960 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:17.510412 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:17.510380 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:25:17.531291 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:17.531262 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:17.700675 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:17.700644 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:18.119192 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:18.119157 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:18.206230 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:18.206192 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:18.307708 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:18.307674 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:18.437744 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:18.437637 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:23.186900 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:23.186864 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:25:27.305591 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:27.305559 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:25:31.635188 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:31.635148 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" podUID="1aeb2e01-1794-43b9-b3fa-e93c29187964" containerName="main" containerID="cri-o://d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a" gracePeriod=45 Jul 06 18:25:31.884113 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:31.884089 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:25:32.020392 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020308 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020392 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020370 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdvwv\" (UniqueName: \"kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020392 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020390 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020406 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020420 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020444 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir\") pod \"1aeb2e01-1794-43b9-b3fa-e93c29187964\" (UID: \"1aeb2e01-1794-43b9-b3fa-e93c29187964\") " Jul 06 18:25:32.020679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020617 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache" (OuterVolumeSpecName: "model-cache") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.020891 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020766 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home" (OuterVolumeSpecName: "home") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.020891 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020789 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.020891 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.020850 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.022733 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.022702 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm" (OuterVolumeSpecName: "dshm") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.022819 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.022747 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv" (OuterVolumeSpecName: "kube-api-access-rdvwv") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "kube-api-access-rdvwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:25:32.022856 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.022816 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1aeb2e01-1794-43b9-b3fa-e93c29187964" (UID: "1aeb2e01-1794-43b9-b3fa-e93c29187964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:25:32.122173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.122135 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.122173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.122167 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aeb2e01-1794-43b9-b3fa-e93c29187964-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.122173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.122178 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.122173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.122185 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeb2e01-1794-43b9-b3fa-e93c29187964-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.122446 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.122195 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvwv\" (UniqueName: \"kubernetes.io/projected/1aeb2e01-1794-43b9-b3fa-e93c29187964-kube-api-access-rdvwv\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.252032 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.251996 2587 generic.go:356] "Generic (PLEG): container finished" podID="1aeb2e01-1794-43b9-b3fa-e93c29187964" containerID="d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a" exitCode=0 Jul 06 18:25:32.252205 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.252043 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" Jul 06 18:25:32.252205 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.252075 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" event={"ID":"1aeb2e01-1794-43b9-b3fa-e93c29187964","Type":"ContainerDied","Data":"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a"} Jul 06 18:25:32.252205 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.252121 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8" event={"ID":"1aeb2e01-1794-43b9-b3fa-e93c29187964","Type":"ContainerDied","Data":"dfec70c14566445dd05f98dd061d001f262e008147d72b32fe045ecb851c2968"} Jul 06 18:25:32.252205 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.252139 2587 scope.go:117] "RemoveContainer" containerID="d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a" Jul 06 18:25:32.261555 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.261532 2587 scope.go:117] "RemoveContainer" containerID="d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a" Jul 06 18:25:32.261868 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:25:32.261849 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a\": container with ID starting with d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a not found: ID does not exist" containerID="d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a" Jul 06 18:25:32.261949 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.261877 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a"} err="failed to get container status \"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a\": rpc error: code = NotFound desc = could not find container \"d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a\": container with ID starting with d63842214cfbab09b51e4e88711154e216ff6bcf81df2a69b17cdffa4bbac50a not found: ID does not exist" Jul 06 18:25:32.283589 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.283506 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:25:32.288508 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.288482 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a9cadd/test-model-a9cadd-kserve-845ddfcd49-qw8h8"] Jul 06 18:25:32.530756 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.530686 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" podUID="059edaf5-4284-4f51-8f3a-0b21832c6592" containerName="main" containerID="cri-o://4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825" gracePeriod=45 Jul 06 18:25:32.785392 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.785332 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:32.929196 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929165 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929196 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929204 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929280 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929297 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929318 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929353 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs\") pod \"059edaf5-4284-4f51-8f3a-0b21832c6592\" (UID: \"059edaf5-4284-4f51-8f3a-0b21832c6592\") " Jul 06 18:25:32.929660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929501 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache" (OuterVolumeSpecName: "model-cache") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.929660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929550 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home" (OuterVolumeSpecName: "home") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.929660 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929593 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.929832 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929687 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.929832 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929708 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.929832 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.929748 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:32.931551 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.931529 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm" (OuterVolumeSpecName: "dshm") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:25:32.931629 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.931545 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47" (OuterVolumeSpecName: "kube-api-access-mbv47") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "kube-api-access-mbv47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:25:32.931629 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:32.931547 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "059edaf5-4284-4f51-8f3a-0b21832c6592" (UID: "059edaf5-4284-4f51-8f3a-0b21832c6592"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:25:33.030935 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.030887 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/059edaf5-4284-4f51-8f3a-0b21832c6592-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:33.030935 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.030931 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/059edaf5-4284-4f51-8f3a-0b21832c6592-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:33.030935 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.030943 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/059edaf5-4284-4f51-8f3a-0b21832c6592-kube-api-access-mbv47\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:25:33.076911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.076880 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aeb2e01-1794-43b9-b3fa-e93c29187964" path="/var/lib/kubelet/pods/1aeb2e01-1794-43b9-b3fa-e93c29187964/volumes" Jul 06 18:25:33.259613 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.259580 2587 generic.go:356] "Generic (PLEG): container finished" podID="059edaf5-4284-4f51-8f3a-0b21832c6592" containerID="4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825" exitCode=0 Jul 06 18:25:33.259781 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.259621 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" Jul 06 18:25:33.259781 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.259642 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" event={"ID":"059edaf5-4284-4f51-8f3a-0b21832c6592","Type":"ContainerDied","Data":"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825"} Jul 06 18:25:33.259781 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.259676 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l" event={"ID":"059edaf5-4284-4f51-8f3a-0b21832c6592","Type":"ContainerDied","Data":"e6e18fc359f99d12e072a1d3e036f8d62dad69614eb36766f024dd06ab3a7a2c"} Jul 06 18:25:33.259781 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.259694 2587 scope.go:117] "RemoveContainer" containerID="4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825" Jul 06 18:25:33.268272 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.268253 2587 scope.go:117] "RemoveContainer" containerID="4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825" Jul 06 18:25:33.268549 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:25:33.268529 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825\": container with ID starting with 4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825 not found: ID does not exist" containerID="4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825" Jul 06 18:25:33.268606 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.268556 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825"} err="failed to get container status \"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825\": rpc error: code = NotFound desc = could not find container \"4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825\": container with ID starting with 4231fd723e14d62bf38b761c4c047f242226b15dc34f0b0e9d6377034a59f825 not found: ID does not exist" Jul 06 18:25:33.307668 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.307637 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:25:33.320584 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:33.315596 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cd9691/test-model-cd9691-kserve-6bb9b96f45-m6f7l"] Jul 06 18:25:35.076700 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:25:35.076666 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="059edaf5-4284-4f51-8f3a-0b21832c6592" path="/var/lib/kubelet/pods/059edaf5-4284-4f51-8f3a-0b21832c6592/volumes" Jul 06 18:26:00.832991 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:00.832957 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:01.121989 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:01.121888 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:01.371433 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:01.371399 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:01.640353 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:01.640315 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:02.078908 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:02.078877 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:02.200320 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:02.200291 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:02.306137 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:02.306105 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:02.516304 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:02.516221 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:05.102217 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:05.102185 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:26:08.407320 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:08.407286 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:08.507708 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:08.507673 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:08.679347 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:08.679270 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:08.908434 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:08.908401 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:09.282937 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:09.282903 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:09.618993 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:09.618959 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:09.868630 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:09.868598 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:11.122209 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:11.122180 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:11.221318 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:11.221288 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:12.316354 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:12.316323 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:12.511388 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:12.511359 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:12.628422 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:12.628318 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:13.249295 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:13.249260 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:13.340308 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:13.340274 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:16.706217 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:16.706138 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:17.201555 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:17.201517 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:17.497230 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:17.497146 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:17.857121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:17.857086 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:19.305381 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:19.305343 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:26:20.294898 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:20.294863 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:20.399500 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:20.399465 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:20.483925 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:20.483896 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:20.611121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:20.611086 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:21.027268 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:21.027181 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:21.137416 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:21.137385 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:24.700489 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:24.700454 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:24.989639 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:24.989551 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:32.784539 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:32.784509 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:33.155668 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:33.155633 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:33.258452 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:33.258413 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:33.358704 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:33.358673 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:33.507558 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:33.507470 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:34.598360 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:34.598320 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:26:39.467588 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:39.467553 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:39.541220 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:39.541190 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:39.755392 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:39.755309 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:39.958357 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:39.958324 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:40.128738 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:40.128690 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:40.521904 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:40.521806 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:40.623320 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:40.623284 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:40.724294 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:40.724260 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:40.891812 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:40.891779 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:41.064351 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:41.064320 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:41.135506 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:41.135464 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:43.682325 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:43.682293 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:43.783411 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:43.783376 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:45.799735 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:45.796981 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:26:47.038162 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:47.038130 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:47.110339 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:47.110307 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:47.373108 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:47.373074 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:47.594249 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:47.594215 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:47.926061 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:47.926025 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:48.255020 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:48.254931 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:48.499313 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:48.499278 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:51.349355 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:51.349324 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:51.439595 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:51.439560 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:51.514600 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:51.514569 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:51.874145 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:51.874115 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:52.034640 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:52.034609 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:52.130598 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:52.130519 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:53.129905 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:53.129876 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:26:53.143193 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:53.143170 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:26:55.840615 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:55.840581 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:56.198243 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:56.198147 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:56.496658 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:56.496573 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:56.619056 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:56.619020 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:57.009758 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:57.009708 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:26:59.403130 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:59.403094 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:59.493397 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:59.493364 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:59.577792 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:59.577763 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:26:59.995529 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:26:59.995490 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:00.096100 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:00.096068 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:00.197273 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:00.197244 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:03.884064 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:03.884023 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:12.404528 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:12.404498 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:27:12.866035 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:12.866004 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:12.968905 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:12.968876 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:20.736400 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:20.736359 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:20.856147 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:20.856118 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:26.059760 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.059709 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:27:26.060151 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060078 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aeb2e01-1794-43b9-b3fa-e93c29187964" containerName="main" Jul 06 18:27:26.060151 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060089 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aeb2e01-1794-43b9-b3fa-e93c29187964" containerName="main" Jul 06 18:27:26.060151 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060112 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="059edaf5-4284-4f51-8f3a-0b21832c6592" containerName="main" Jul 06 18:27:26.060151 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060118 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="059edaf5-4284-4f51-8f3a-0b21832c6592" containerName="main" Jul 06 18:27:26.060286 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060172 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aeb2e01-1794-43b9-b3fa-e93c29187964" containerName="main" Jul 06 18:27:26.060286 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.060180 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="059edaf5-4284-4f51-8f3a-0b21832c6592" containerName="main" Jul 06 18:27:26.063096 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.063081 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.068744 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.068701 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"openshift-service-ca.crt\"" Jul 06 18:27:26.068744 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.068738 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"kube-root-ca.crt\"" Jul 06 18:27:26.068922 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.068746 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"default-dockercfg-mrm5w\"" Jul 06 18:27:26.068922 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.068815 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"auth-test-model-9c9be0-kserve-self-signed-certs\"" Jul 06 18:27:26.091613 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.091587 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:27:26.118423 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.118389 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:26.197045 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197009 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.197247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197054 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.197247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197073 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.197247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197100 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.197247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197152 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2ln\" (UniqueName: \"kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.197247 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.197223 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298264 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298224 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298278 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298303 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298327 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298355 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2ln\" (UniqueName: \"kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298407 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298790 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298686 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298839 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298815 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.298875 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.298840 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.300678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.300655 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.300886 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.300868 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.310436 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.310371 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2ln\" (UniqueName: \"kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln\") pod \"auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.372203 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.372171 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:26.507092 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.507067 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:27:26.508903 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:27:26.508874 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd28fa0e7_c947_45e9_b93d_53e6eb0fbe7a.slice/crio-f8e90e35916cc04fe0464dc5dd2345fe140187ae5377223b5a32818be15dea96 WatchSource:0}: Error finding container f8e90e35916cc04fe0464dc5dd2345fe140187ae5377223b5a32818be15dea96: Status 404 returned error can't find the container with id f8e90e35916cc04fe0464dc5dd2345fe140187ae5377223b5a32818be15dea96 Jul 06 18:27:26.859336 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.859240 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" event={"ID":"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a","Type":"ContainerStarted","Data":"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472"} Jul 06 18:27:26.859336 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.859280 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" event={"ID":"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a","Type":"ContainerStarted","Data":"f8e90e35916cc04fe0464dc5dd2345fe140187ae5377223b5a32818be15dea96"} Jul 06 18:27:26.878864 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:26.878801 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" podStartSLOduration=0.878785713 podStartE2EDuration="878.785713ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:27:26.877912228 +0000 UTC m=+2134.407444625" watchObservedRunningTime="2026-07-06 18:27:26.878785713 +0000 UTC m=+2134.408318073" Jul 06 18:27:29.197999 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:29.197961 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:27:36.372976 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:36.372942 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:36.373370 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:36.373081 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:36.385337 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:36.385314 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:36.923065 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:36.923037 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:27:51.108537 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.108462 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:27:51.111863 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.111844 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.115552 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.115525 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"auth-test-model-b6a5ee-kserve-self-signed-certs\"" Jul 06 18:27:51.115668 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.115569 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"openshift-service-ca.crt\"" Jul 06 18:27:51.115668 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.115652 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"kube-root-ca.crt\"" Jul 06 18:27:51.115822 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.115659 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"default-dockercfg-7zf4w\"" Jul 06 18:27:51.120638 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.120618 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:27:51.130624 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.130593 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:27:51.209207 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209176 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.209207 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209208 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.209426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209232 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.209426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209264 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr85\" (UniqueName: \"kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.209426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209326 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.209426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.209359 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.310847 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.310804 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.310847 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.310853 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.310946 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.310967 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.311001 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311121 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.311035 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr85\" (UniqueName: \"kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311318 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.311253 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311393 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.311317 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.311393 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.311267 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.313284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.313262 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.313464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.313445 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.319973 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.319946 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr85\" (UniqueName: \"kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85\") pod \"auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.422300 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.422208 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:27:51.552962 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.552938 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:27:51.555226 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:27:51.555179 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podace9a2aa_3adb_4319_868c_8f081815ab57.slice/crio-39a444f14585ee8b5dc84e0d6d535b878634d111c287e51320efcde966f6ece7 WatchSource:0}: Error finding container 39a444f14585ee8b5dc84e0d6d535b878634d111c287e51320efcde966f6ece7: Status 404 returned error can't find the container with id 39a444f14585ee8b5dc84e0d6d535b878634d111c287e51320efcde966f6ece7 Jul 06 18:27:51.989320 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.989284 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" event={"ID":"ace9a2aa-3adb-4319-868c-8f081815ab57","Type":"ContainerStarted","Data":"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088"} Jul 06 18:27:51.989501 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:51.989333 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" event={"ID":"ace9a2aa-3adb-4319-868c-8f081815ab57","Type":"ContainerStarted","Data":"39a444f14585ee8b5dc84e0d6d535b878634d111c287e51320efcde966f6ece7"} Jul 06 18:27:52.007880 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:52.007831 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" podStartSLOduration=1.007815633 podStartE2EDuration="1.007815633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:27:52.006324307 +0000 UTC m=+2159.535856667" watchObservedRunningTime="2026-07-06 18:27:52.007815633 +0000 UTC m=+2159.537348003" Jul 06 18:27:54.694511 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:27:54.694473 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:28:01.422594 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:01.422556 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:28:01.423125 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:01.422879 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:28:01.435259 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:01.435232 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:28:02.052035 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:02.052010 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:28:10.296000 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:10.295969 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:28:28.096302 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:28.096266 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:28:45.294423 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:28:45.294392 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:29:00.198953 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:00.198918 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:29:16.295076 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:16.294998 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:29:30.186238 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:30.186201 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:29:39.873490 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:39.873451 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:29:39.924224 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:39.924189 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:29:40.801024 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:40.800991 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:29:40.820492 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:40.820457 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:29:45.189066 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:45.189026 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:29:46.299010 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.298981 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:29:46.302419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.302400 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.304895 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.304875 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"sub-test-model-9c9be0-kserve-self-signed-certs\"" Jul 06 18:29:46.310598 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.310577 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:29:46.315412 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.315385 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:29:46.446396 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446357 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.446396 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446397 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.446630 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446415 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.446630 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446430 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.446630 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446460 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtww\" (UniqueName: \"kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.446630 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.446494 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547238 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547205 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547400 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547380 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547428 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547483 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547460 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547518 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547483 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547518 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547508 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtww\" (UniqueName: \"kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547664 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547648 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547740 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547696 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.547847 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.547827 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.549845 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.549822 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.550128 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.550107 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.556544 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.556522 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtww\" (UniqueName: \"kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww\") pod \"sub-test-model-9c9be0-kserve-6f6f86859-d96z6\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.613168 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.613149 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:46.745124 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:46.745098 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:29:46.747304 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:29:46.747275 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bad8a90_1357_4309_880b_ebb9bf0700e7.slice/crio-ccbe2eb07fc817eb6f7f60f436580686af24e676db6e51f6cdc6854e8a9b5e19 WatchSource:0}: Error finding container ccbe2eb07fc817eb6f7f60f436580686af24e676db6e51f6cdc6854e8a9b5e19: Status 404 returned error can't find the container with id ccbe2eb07fc817eb6f7f60f436580686af24e676db6e51f6cdc6854e8a9b5e19 Jul 06 18:29:47.589592 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:47.589557 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" event={"ID":"4bad8a90-1357-4309-880b-ebb9bf0700e7","Type":"ContainerStarted","Data":"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6"} Jul 06 18:29:47.589592 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:47.589597 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" event={"ID":"4bad8a90-1357-4309-880b-ebb9bf0700e7","Type":"ContainerStarted","Data":"ccbe2eb07fc817eb6f7f60f436580686af24e676db6e51f6cdc6854e8a9b5e19"} Jul 06 18:29:47.608431 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:47.608384 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" podStartSLOduration=1.6083700269999999 podStartE2EDuration="1.608370027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:29:47.608207039 +0000 UTC m=+2275.137739398" watchObservedRunningTime="2026-07-06 18:29:47.608370027 +0000 UTC m=+2275.137902385" Jul 06 18:29:54.893140 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:54.893079 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" podUID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" containerName="main" containerID="cri-o://370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472" gracePeriod=45 Jul 06 18:29:55.136346 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.136324 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:29:55.228765 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228678 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228765 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228706 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228994 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228798 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228994 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228832 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv2ln\" (UniqueName: \"kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228994 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228862 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228994 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228918 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home\") pod \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\" (UID: \"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a\") " Jul 06 18:29:55.228994 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.228916 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:55.229254 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.229145 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:55.229254 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.229166 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home" (OuterVolumeSpecName: "home") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:55.229254 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.229198 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.231096 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.231073 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:29:55.231096 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.231091 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm" (OuterVolumeSpecName: "dshm") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:55.231258 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.231105 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln" (OuterVolumeSpecName: "kube-api-access-bv2ln") pod "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" (UID: "d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a"). InnerVolumeSpecName "kube-api-access-bv2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:29:55.330288 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.330261 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2ln\" (UniqueName: \"kubernetes.io/projected/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-kube-api-access-bv2ln\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.330288 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.330286 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.330288 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.330295 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.330514 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.330303 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.330514 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.330312 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:55.635535 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.635503 2587 generic.go:356] "Generic (PLEG): container finished" podID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" containerID="370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472" exitCode=0 Jul 06 18:29:55.635535 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.635542 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" Jul 06 18:29:55.635835 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.635556 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" event={"ID":"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a","Type":"ContainerDied","Data":"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472"} Jul 06 18:29:55.635835 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.635594 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj" event={"ID":"d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a","Type":"ContainerDied","Data":"f8e90e35916cc04fe0464dc5dd2345fe140187ae5377223b5a32818be15dea96"} Jul 06 18:29:55.635835 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.635619 2587 scope.go:117] "RemoveContainer" containerID="370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472" Jul 06 18:29:55.654282 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.654256 2587 scope.go:117] "RemoveContainer" containerID="370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472" Jul 06 18:29:55.654642 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:29:55.654618 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472\": container with ID starting with 370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472 not found: ID does not exist" containerID="370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472" Jul 06 18:29:55.654731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.654651 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472"} err="failed to get container status \"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472\": rpc error: code = NotFound desc = could not find container \"370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472\": container with ID starting with 370d5f94bc7a3d3d1682ad821f03b2835b22bf1491cef04a065393c76d846472 not found: ID does not exist" Jul 06 18:29:55.680254 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.680224 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:29:55.690389 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.690362 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/auth-test-model-9c9be0-kserve-5c968f5d7-xhjzj"] Jul 06 18:29:55.821082 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:55.821037 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" podUID="ace9a2aa-3adb-4319-868c-8f081815ab57" containerName="main" containerID="cri-o://fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088" gracePeriod=45 Jul 06 18:29:56.073060 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.073038 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:29:56.238904 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.238811 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.238904 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.238846 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.238904 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.238874 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.239177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.238926 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.239177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.238974 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbr85\" (UniqueName: \"kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.239177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239035 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home\") pod \"ace9a2aa-3adb-4319-868c-8f081815ab57\" (UID: \"ace9a2aa-3adb-4319-868c-8f081815ab57\") " Jul 06 18:29:56.239177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239129 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:56.239177 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239156 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache" (OuterVolumeSpecName: "model-cache") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:56.239419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239384 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.239470 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239432 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.239534 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.239498 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home" (OuterVolumeSpecName: "home") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:56.241179 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.241158 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm" (OuterVolumeSpecName: "dshm") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:29:56.241259 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.241202 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:29:56.241304 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.241282 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85" (OuterVolumeSpecName: "kube-api-access-bbr85") pod "ace9a2aa-3adb-4319-868c-8f081815ab57" (UID: "ace9a2aa-3adb-4319-868c-8f081815ab57"). InnerVolumeSpecName "kube-api-access-bbr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:29:56.340321 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.340269 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace9a2aa-3adb-4319-868c-8f081815ab57-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.340321 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.340321 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbr85\" (UniqueName: \"kubernetes.io/projected/ace9a2aa-3adb-4319-868c-8f081815ab57-kube-api-access-bbr85\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.340321 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.340331 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.340509 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.340339 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace9a2aa-3adb-4319-868c-8f081815ab57-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:29:56.614078 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.614051 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:56.614216 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.614095 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:56.625893 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.625871 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:56.641625 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.641601 2587 generic.go:356] "Generic (PLEG): container finished" podID="ace9a2aa-3adb-4319-868c-8f081815ab57" containerID="fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088" exitCode=0 Jul 06 18:29:56.641753 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.641649 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" Jul 06 18:29:56.641753 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.641684 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" event={"ID":"ace9a2aa-3adb-4319-868c-8f081815ab57","Type":"ContainerDied","Data":"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088"} Jul 06 18:29:56.641838 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.641752 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj" event={"ID":"ace9a2aa-3adb-4319-868c-8f081815ab57","Type":"ContainerDied","Data":"39a444f14585ee8b5dc84e0d6d535b878634d111c287e51320efcde966f6ece7"} Jul 06 18:29:56.641838 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.641772 2587 scope.go:117] "RemoveContainer" containerID="fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088" Jul 06 18:29:56.651878 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.651860 2587 scope.go:117] "RemoveContainer" containerID="fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088" Jul 06 18:29:56.652115 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:29:56.652096 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088\": container with ID starting with fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088 not found: ID does not exist" containerID="fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088" Jul 06 18:29:56.652179 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.652126 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088"} err="failed to get container status \"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088\": rpc error: code = NotFound desc = could not find container \"fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088\": container with ID starting with fa41b8df7d7987dfd773f68783a7637ecf1a8cf01b0142b8886a086269551088 not found: ID does not exist" Jul 06 18:29:56.655424 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.655398 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:29:56.665161 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.665142 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:29:56.670955 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:56.670938 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/auth-test-model-b6a5ee-kserve-85dccf6f55-2f9cj"] Jul 06 18:29:57.076179 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:57.076149 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace9a2aa-3adb-4319-868c-8f081815ab57" path="/var/lib/kubelet/pods/ace9a2aa-3adb-4319-868c-8f081815ab57/volumes" Jul 06 18:29:57.076525 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:57.076518 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" path="/var/lib/kubelet/pods/d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a/volumes" Jul 06 18:29:59.890311 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:29:59.890273 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:30:00.133154 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133121 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv"] Jul 06 18:30:00.133495 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133476 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" containerName="main" Jul 06 18:30:00.133593 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133499 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" containerName="main" Jul 06 18:30:00.133593 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133526 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace9a2aa-3adb-4319-868c-8f081815ab57" containerName="main" Jul 06 18:30:00.133593 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133534 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace9a2aa-3adb-4319-868c-8f081815ab57" containerName="main" Jul 06 18:30:00.133800 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133645 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace9a2aa-3adb-4319-868c-8f081815ab57" containerName="main" Jul 06 18:30:00.133800 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.133656 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28fa0e7-c947-45e9-b93d-53e6eb0fbe7a" containerName="main" Jul 06 18:30:00.136749 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.136729 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct"] Jul 06 18:30:00.136846 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.136824 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:00.139221 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.139205 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-wkkbb\"" Jul 06 18:30:00.139853 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.139839 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722710-lgnpj"] Jul 06 18:30:00.139929 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.139897 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:00.142869 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.142807 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:00.145907 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.145887 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv"] Jul 06 18:30:00.149047 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.149024 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct"] Jul 06 18:30:00.152063 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.152043 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722710-lgnpj"] Jul 06 18:30:00.275248 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.275208 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p6n\" (UniqueName: \"kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n\") pod \"maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct\" (UID: \"73426edf-36bb-485f-a945-b53d8c2f79d9\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:00.275434 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.275259 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftq4\" (UniqueName: \"kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4\") pod \"maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv\" (UID: \"ad9f0401-5cc4-454f-9232-ce2768e7e9fb\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:00.275434 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.275348 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vp6\" (UniqueName: \"kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6\") pod \"maas-api-key-cleanup-29722710-lgnpj\" (UID: \"0962c026-223a-4766-a429-f236797f7218\") " pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:00.376098 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.376059 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p6n\" (UniqueName: \"kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n\") pod \"maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct\" (UID: \"73426edf-36bb-485f-a945-b53d8c2f79d9\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:00.376269 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.376110 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftq4\" (UniqueName: \"kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4\") pod \"maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv\" (UID: \"ad9f0401-5cc4-454f-9232-ce2768e7e9fb\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:00.376269 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.376158 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vp6\" (UniqueName: \"kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6\") pod \"maas-api-key-cleanup-29722710-lgnpj\" (UID: \"0962c026-223a-4766-a429-f236797f7218\") " pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:00.385013 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.384985 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vp6\" (UniqueName: \"kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6\") pod \"maas-api-key-cleanup-29722710-lgnpj\" (UID: \"0962c026-223a-4766-a429-f236797f7218\") " pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:00.385141 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.385119 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p6n\" (UniqueName: \"kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n\") pod \"maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct\" (UID: \"73426edf-36bb-485f-a945-b53d8c2f79d9\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:00.385183 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.385119 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftq4\" (UniqueName: \"kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4\") pod \"maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv\" (UID: \"ad9f0401-5cc4-454f-9232-ce2768e7e9fb\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:00.450447 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.450374 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:00.465543 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.465518 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:00.471545 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.471528 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:00.817526 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.817484 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv"] Jul 06 18:30:00.818642 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:30:00.818616 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9f0401_5cc4_454f_9232_ce2768e7e9fb.slice/crio-de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87 WatchSource:0}: Error finding container de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87: Status 404 returned error can't find the container with id de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87 Jul 06 18:30:00.831328 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.831300 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct"] Jul 06 18:30:00.833283 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:30:00.833246 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73426edf_36bb_485f_a945_b53d8c2f79d9.slice/crio-733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1 WatchSource:0}: Error finding container 733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1: Status 404 returned error can't find the container with id 733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1 Jul 06 18:30:00.835187 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:00.835170 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722710-lgnpj"] Jul 06 18:30:00.836944 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:30:00.836918 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0962c026_223a_4766_a429_f236797f7218.slice/crio-092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010 WatchSource:0}: Error finding container 092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010: Status 404 returned error can't find the container with id 092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010 Jul 06 18:30:01.670383 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.670340 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" event={"ID":"0962c026-223a-4766-a429-f236797f7218","Type":"ContainerStarted","Data":"d7926eee4d8148f140e0513d27c1098daa7f64fe1606314984bed7f3332d6bf4"} Jul 06 18:30:01.670946 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.670405 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" event={"ID":"0962c026-223a-4766-a429-f236797f7218","Type":"ContainerStarted","Data":"092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010"} Jul 06 18:30:01.671786 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.671759 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" event={"ID":"73426edf-36bb-485f-a945-b53d8c2f79d9","Type":"ContainerStarted","Data":"d64c2f36dfa5428f8f611aa44ed6e47a8631dbc5cd4ded92e53ba819b56871e4"} Jul 06 18:30:01.671901 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.671797 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" event={"ID":"73426edf-36bb-485f-a945-b53d8c2f79d9","Type":"ContainerStarted","Data":"733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1"} Jul 06 18:30:01.673074 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.673055 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" event={"ID":"ad9f0401-5cc4-454f-9232-ce2768e7e9fb","Type":"ContainerStarted","Data":"2ce39fa3236e234c48ba74641ba27ee1279a38d3596daa6cec2f99ac84d211ee"} Jul 06 18:30:01.673174 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.673092 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" event={"ID":"ad9f0401-5cc4-454f-9232-ce2768e7e9fb","Type":"ContainerStarted","Data":"de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87"} Jul 06 18:30:01.685917 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.685869 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" podStartSLOduration=1.68585751 podStartE2EDuration="1.68585751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:30:01.684941826 +0000 UTC m=+2289.214474186" watchObservedRunningTime="2026-07-06 18:30:01.68585751 +0000 UTC m=+2289.215389917" Jul 06 18:30:01.701172 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.701128 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" podStartSLOduration=1.701115826 podStartE2EDuration="1.701115826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:30:01.699097895 +0000 UTC m=+2289.228630277" watchObservedRunningTime="2026-07-06 18:30:01.701115826 +0000 UTC m=+2289.230648185" Jul 06 18:30:01.714544 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:01.714508 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" podStartSLOduration=1.714492556 podStartE2EDuration="1.714492556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:30:01.712593917 +0000 UTC m=+2289.242126274" watchObservedRunningTime="2026-07-06 18:30:01.714492556 +0000 UTC m=+2289.244024917" Jul 06 18:30:02.680196 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.680157 2587 generic.go:356] "Generic (PLEG): container finished" podID="0962c026-223a-4766-a429-f236797f7218" containerID="d7926eee4d8148f140e0513d27c1098daa7f64fe1606314984bed7f3332d6bf4" exitCode=0 Jul 06 18:30:02.680667 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.680246 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" event={"ID":"0962c026-223a-4766-a429-f236797f7218","Type":"ContainerDied","Data":"d7926eee4d8148f140e0513d27c1098daa7f64fe1606314984bed7f3332d6bf4"} Jul 06 18:30:02.681543 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.681518 2587 generic.go:356] "Generic (PLEG): container finished" podID="73426edf-36bb-485f-a945-b53d8c2f79d9" containerID="d64c2f36dfa5428f8f611aa44ed6e47a8631dbc5cd4ded92e53ba819b56871e4" exitCode=0 Jul 06 18:30:02.681694 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.681586 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" event={"ID":"73426edf-36bb-485f-a945-b53d8c2f79d9","Type":"ContainerDied","Data":"d64c2f36dfa5428f8f611aa44ed6e47a8631dbc5cd4ded92e53ba819b56871e4"} Jul 06 18:30:02.682834 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.682812 2587 generic.go:356] "Generic (PLEG): container finished" podID="ad9f0401-5cc4-454f-9232-ce2768e7e9fb" containerID="2ce39fa3236e234c48ba74641ba27ee1279a38d3596daa6cec2f99ac84d211ee" exitCode=0 Jul 06 18:30:02.682954 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:02.682893 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" event={"ID":"ad9f0401-5cc4-454f-9232-ce2768e7e9fb","Type":"ContainerDied","Data":"2ce39fa3236e234c48ba74641ba27ee1279a38d3596daa6cec2f99ac84d211ee"} Jul 06 18:30:03.832206 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:03.832179 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:03.884009 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:03.883983 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:03.886846 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:03.886828 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:04.009991 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.009903 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96p6n\" (UniqueName: \"kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n\") pod \"73426edf-36bb-485f-a945-b53d8c2f79d9\" (UID: \"73426edf-36bb-485f-a945-b53d8c2f79d9\") " Jul 06 18:30:04.010142 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.009995 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hftq4\" (UniqueName: \"kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4\") pod \"ad9f0401-5cc4-454f-9232-ce2768e7e9fb\" (UID: \"ad9f0401-5cc4-454f-9232-ce2768e7e9fb\") " Jul 06 18:30:04.010142 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.010033 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6vp6\" (UniqueName: \"kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6\") pod \"0962c026-223a-4766-a429-f236797f7218\" (UID: \"0962c026-223a-4766-a429-f236797f7218\") " Jul 06 18:30:04.012144 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.012115 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n" (OuterVolumeSpecName: "kube-api-access-96p6n") pod "73426edf-36bb-485f-a945-b53d8c2f79d9" (UID: "73426edf-36bb-485f-a945-b53d8c2f79d9"). InnerVolumeSpecName "kube-api-access-96p6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:30:04.012144 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.012134 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4" (OuterVolumeSpecName: "kube-api-access-hftq4") pod "ad9f0401-5cc4-454f-9232-ce2768e7e9fb" (UID: "ad9f0401-5cc4-454f-9232-ce2768e7e9fb"). InnerVolumeSpecName "kube-api-access-hftq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:30:04.012286 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.012123 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6" (OuterVolumeSpecName: "kube-api-access-j6vp6") pod "0962c026-223a-4766-a429-f236797f7218" (UID: "0962c026-223a-4766-a429-f236797f7218"). InnerVolumeSpecName "kube-api-access-j6vp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:30:04.110865 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.110826 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vp6\" (UniqueName: \"kubernetes.io/projected/0962c026-223a-4766-a429-f236797f7218-kube-api-access-j6vp6\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:04.110865 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.110860 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96p6n\" (UniqueName: \"kubernetes.io/projected/73426edf-36bb-485f-a945-b53d8c2f79d9-kube-api-access-96p6n\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:04.110865 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.110872 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hftq4\" (UniqueName: \"kubernetes.io/projected/ad9f0401-5cc4-454f-9232-ce2768e7e9fb-kube-api-access-hftq4\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:04.695241 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.695210 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" Jul 06 18:30:04.695241 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.695222 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722710-lgnpj" event={"ID":"0962c026-223a-4766-a429-f236797f7218","Type":"ContainerDied","Data":"092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010"} Jul 06 18:30:04.695489 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.695293 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092e52c415d841cce17ee5123cb964672d99d1dd7c4a10657725863ecfdac010" Jul 06 18:30:04.696352 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.696333 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" Jul 06 18:30:04.696475 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.696331 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct" event={"ID":"73426edf-36bb-485f-a945-b53d8c2f79d9","Type":"ContainerDied","Data":"733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1"} Jul 06 18:30:04.696475 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.696434 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733da4dfa6e40a6336a468fc8666b7d1e643c0fff094cad5cb79c31bc302fde1" Jul 06 18:30:04.697583 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.697561 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" Jul 06 18:30:04.697731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.697600 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv" event={"ID":"ad9f0401-5cc4-454f-9232-ce2768e7e9fb","Type":"ContainerDied","Data":"de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87"} Jul 06 18:30:04.697731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:04.697645 2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de88f15d16f7f5dd6a2d3de3bf2b9bdb06030145da01a331680d17bf2338ce87" Jul 06 18:30:10.189408 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189377 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189749 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73426edf-36bb-485f-a945-b53d8c2f79d9" containerName="cleanup" Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189761 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="73426edf-36bb-485f-a945-b53d8c2f79d9" containerName="cleanup" Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189773 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9f0401-5cc4-454f-9232-ce2768e7e9fb" containerName="cleanup" Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189778 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9f0401-5cc4-454f-9232-ce2768e7e9fb" containerName="cleanup" Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189790 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0962c026-223a-4766-a429-f236797f7218" containerName="cleanup" Jul 06 18:30:10.189816 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189797 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="0962c026-223a-4766-a429-f236797f7218" containerName="cleanup" Jul 06 18:30:10.190007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189861 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="0962c026-223a-4766-a429-f236797f7218" containerName="cleanup" Jul 06 18:30:10.190007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189867 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9f0401-5cc4-454f-9232-ce2768e7e9fb" containerName="cleanup" Jul 06 18:30:10.190007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.189874 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="73426edf-36bb-485f-a945-b53d8c2f79d9" containerName="cleanup" Jul 06 18:30:10.193034 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.193018 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.200562 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.200531 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"kube-root-ca.crt\"" Jul 06 18:30:10.201730 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.201660 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"openshift-service-ca.crt\"" Jul 06 18:30:10.201863 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.201780 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"default-dockercfg-7zf4w\"" Jul 06 18:30:10.201863 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.201780 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"sub-test-model-b6a5ee-kserve-self-signed-certs\"" Jul 06 18:30:10.205161 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.205139 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:10.225323 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.225291 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:10.261919 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.261885 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.261919 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.261921 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.262142 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.261997 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.262142 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.262041 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csbb\" (UniqueName: \"kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.262142 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.262096 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.262258 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.262144 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.362913 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.362873 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.362913 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.362916 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363155 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.362966 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363155 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.362988 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csbb\" (UniqueName: \"kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363155 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.363019 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363155 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.363055 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363362 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.363304 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363469 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.363442 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.363580 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.363486 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.365284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.365265 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.365556 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.365537 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.371122 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.371101 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csbb\" (UniqueName: \"kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb\") pod \"sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.509174 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.509084 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:10.641741 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.641695 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:10.644234 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:30:10.644210 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a54e77a_8c28_4bfb_9112_7d935cb6a614.slice/crio-9a8121ef5e56a82281eae4ac98895c4fe08ac3d65c605bcda9c89e086c6d4333 WatchSource:0}: Error finding container 9a8121ef5e56a82281eae4ac98895c4fe08ac3d65c605bcda9c89e086c6d4333: Status 404 returned error can't find the container with id 9a8121ef5e56a82281eae4ac98895c4fe08ac3d65c605bcda9c89e086c6d4333 Jul 06 18:30:10.732945 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.732904 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" event={"ID":"8a54e77a-8c28-4bfb-9112-7d935cb6a614","Type":"ContainerStarted","Data":"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576"} Jul 06 18:30:10.732945 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.732946 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" event={"ID":"8a54e77a-8c28-4bfb-9112-7d935cb6a614","Type":"ContainerStarted","Data":"9a8121ef5e56a82281eae4ac98895c4fe08ac3d65c605bcda9c89e086c6d4333"} Jul 06 18:30:10.755261 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:10.755214 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" podStartSLOduration=0.755199135 podStartE2EDuration="755.199135ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:30:10.753196965 +0000 UTC m=+2298.282729327" watchObservedRunningTime="2026-07-06 18:30:10.755199135 +0000 UTC m=+2298.284731493" Jul 06 18:30:14.285959 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:14.285923 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:30:20.509781 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:20.509744 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:20.510218 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:20.510018 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:20.522211 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:20.522180 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:20.797064 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:20.796975 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:28.193749 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:28.193705 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:30:38.863345 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:38.863306 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:42.050034 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:42.050003 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:30:42.067017 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:42.066986 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:42.981983 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:42.981953 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:43.002302 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:43.002265 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:45.391955 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:45.391863 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:46.388929 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:46.388901 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:30:51.065226 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:51.065188 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:30:57.070102 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.070037 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" podUID="4bad8a90-1357-4309-880b-ebb9bf0700e7" containerName="main" containerID="cri-o://49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6" gracePeriod=45 Jul 06 18:30:57.316170 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.316144 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:30:57.381267 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381167 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381267 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381218 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381267 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381242 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qtww\" (UniqueName: \"kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381562 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381357 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381562 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381437 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381562 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381464 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache\") pod \"4bad8a90-1357-4309-880b-ebb9bf0700e7\" (UID: \"4bad8a90-1357-4309-880b-ebb9bf0700e7\") " Jul 06 18:30:57.381562 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381499 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:57.381802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381670 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home" (OuterVolumeSpecName: "home") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:57.381802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381781 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.381802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381798 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.381941 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.381793 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:57.383395 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.383362 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:30:57.383789 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.383771 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm" (OuterVolumeSpecName: "dshm") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:57.383864 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.383847 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww" (OuterVolumeSpecName: "kube-api-access-8qtww") pod "4bad8a90-1357-4309-880b-ebb9bf0700e7" (UID: "4bad8a90-1357-4309-880b-ebb9bf0700e7"). InnerVolumeSpecName "kube-api-access-8qtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:30:57.482741 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.482681 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.482741 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.482735 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bad8a90-1357-4309-880b-ebb9bf0700e7-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.482741 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.482746 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtww\" (UniqueName: \"kubernetes.io/projected/4bad8a90-1357-4309-880b-ebb9bf0700e7-kube-api-access-8qtww\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.482987 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.482756 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bad8a90-1357-4309-880b-ebb9bf0700e7-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:57.988284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.988251 2587 generic.go:356] "Generic (PLEG): container finished" podID="4bad8a90-1357-4309-880b-ebb9bf0700e7" containerID="49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6" exitCode=0 Jul 06 18:30:57.988284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.988285 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" Jul 06 18:30:57.988540 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.988330 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" event={"ID":"4bad8a90-1357-4309-880b-ebb9bf0700e7","Type":"ContainerDied","Data":"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6"} Jul 06 18:30:57.988540 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.988373 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6" event={"ID":"4bad8a90-1357-4309-880b-ebb9bf0700e7","Type":"ContainerDied","Data":"ccbe2eb07fc817eb6f7f60f436580686af24e676db6e51f6cdc6854e8a9b5e19"} Jul 06 18:30:57.988540 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.988397 2587 scope.go:117] "RemoveContainer" containerID="49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6" Jul 06 18:30:57.998587 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.998566 2587 scope.go:117] "RemoveContainer" containerID="49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6" Jul 06 18:30:57.998921 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:30:57.998900 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6\": container with ID starting with 49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6 not found: ID does not exist" containerID="49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6" Jul 06 18:30:57.999006 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:57.998929 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6"} err="failed to get container status \"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6\": rpc error: code = NotFound desc = could not find container \"49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6\": container with ID starting with 49626aa9ba62a5b4584955346af422349f71746270ef236687389cf7d1ab85e6 not found: ID does not exist" Jul 06 18:30:58.002168 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.002140 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" podUID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" containerName="main" containerID="cri-o://a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576" gracePeriod=45 Jul 06 18:30:58.012780 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.012754 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:30:58.016062 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.016038 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/sub-test-model-9c9be0-kserve-6f6f86859-d96z6"] Jul 06 18:30:58.246802 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.246747 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:58.289270 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289240 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289280 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289298 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289314 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4csbb\" (UniqueName: \"kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289330 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289426 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289357 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home\") pod \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\" (UID: \"8a54e77a-8c28-4bfb-9112-7d935cb6a614\") " Jul 06 18:30:58.289709 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289584 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:58.289709 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289600 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:58.289821 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.289707 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home" (OuterVolumeSpecName: "home") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:58.291444 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.291409 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm" (OuterVolumeSpecName: "dshm") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:30:58.291551 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.291483 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:30:58.291551 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.291501 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb" (OuterVolumeSpecName: "kube-api-access-4csbb") pod "8a54e77a-8c28-4bfb-9112-7d935cb6a614" (UID: "8a54e77a-8c28-4bfb-9112-7d935cb6a614"). InnerVolumeSpecName "kube-api-access-4csbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:30:58.390936 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390901 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.390936 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390930 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a54e77a-8c28-4bfb-9112-7d935cb6a614-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.390936 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390942 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csbb\" (UniqueName: \"kubernetes.io/projected/8a54e77a-8c28-4bfb-9112-7d935cb6a614-kube-api-access-4csbb\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.391190 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390951 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.391190 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390961 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.391190 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.390968 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a54e77a-8c28-4bfb-9112-7d935cb6a614-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:30:58.996538 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.996498 2587 generic.go:356] "Generic (PLEG): container finished" podID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" containerID="a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576" exitCode=0 Jul 06 18:30:58.996697 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.996565 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" Jul 06 18:30:58.996697 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.996584 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" event={"ID":"8a54e77a-8c28-4bfb-9112-7d935cb6a614","Type":"ContainerDied","Data":"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576"} Jul 06 18:30:58.996697 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.996632 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm" event={"ID":"8a54e77a-8c28-4bfb-9112-7d935cb6a614","Type":"ContainerDied","Data":"9a8121ef5e56a82281eae4ac98895c4fe08ac3d65c605bcda9c89e086c6d4333"} Jul 06 18:30:58.996697 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:58.996649 2587 scope.go:117] "RemoveContainer" containerID="a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576" Jul 06 18:30:59.006221 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.006200 2587 scope.go:117] "RemoveContainer" containerID="a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576" Jul 06 18:30:59.006465 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:30:59.006443 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576\": container with ID starting with a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576 not found: ID does not exist" containerID="a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576" Jul 06 18:30:59.006528 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.006477 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576"} err="failed to get container status \"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576\": rpc error: code = NotFound desc = could not find container \"a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576\": container with ID starting with a4442dfb1948a0df1f688cd64063d8239e62d98a2421fab25c9721f969aeb576 not found: ID does not exist" Jul 06 18:30:59.018662 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.018637 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:59.022503 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.022481 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/sub-test-model-b6a5ee-kserve-7f498f949f-t7gjm"] Jul 06 18:30:59.076747 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.076699 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bad8a90-1357-4309-880b-ebb9bf0700e7" path="/var/lib/kubelet/pods/4bad8a90-1357-4309-880b-ebb9bf0700e7/volumes" Jul 06 18:30:59.077122 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:30:59.077108 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" path="/var/lib/kubelet/pods/8a54e77a-8c28-4bfb-9112-7d935cb6a614/volumes" Jul 06 18:31:01.596488 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596460 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:31:01.596857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596813 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bad8a90-1357-4309-880b-ebb9bf0700e7" containerName="main" Jul 06 18:31:01.596857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596825 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bad8a90-1357-4309-880b-ebb9bf0700e7" containerName="main" Jul 06 18:31:01.596857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596838 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" containerName="main" Jul 06 18:31:01.596857 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596844 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" containerName="main" Jul 06 18:31:01.597003 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596908 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a54e77a-8c28-4bfb-9112-7d935cb6a614" containerName="main" Jul 06 18:31:01.597003 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.596917 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bad8a90-1357-4309-880b-ebb9bf0700e7" containerName="main" Jul 06 18:31:01.600908 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.600891 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.604843 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.604820 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"default-dockercfg-mrm5w\"" Jul 06 18:31:01.605002 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.604866 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"kube-root-ca.crt\"" Jul 06 18:31:01.605002 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.604882 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"openshift-service-ca.crt\"" Jul 06 18:31:01.605137 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.605016 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c9be0\"/\"rate-test-model-9c9be0-kserve-self-signed-certs\"" Jul 06 18:31:01.610965 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.610943 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:31:01.616194 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616174 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.616303 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616201 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.616303 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616223 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.616303 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616290 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzb9p\" (UniqueName: \"kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.616465 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616370 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.616465 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.616413 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.646382 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.646345 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:31:01.717640 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717598 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.717640 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717641 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.717942 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717676 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.717942 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717742 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzb9p\" (UniqueName: \"kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.717942 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717780 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.717942 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.717815 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.718225 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.718152 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.718225 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.718191 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.718225 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.718210 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.720070 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.720047 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.720224 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.720208 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.726295 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.726271 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzb9p\" (UniqueName: \"kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p\") pod \"rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:01.911191 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:01.911095 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:02.047104 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:02.046825 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:31:02.050528 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:31:02.050493 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66356428_f379_4021_bdf7_dc4ace50dcea.slice/crio-18a613528543ce7e29c758afd0250ab8cfd93b48912fa8022a75f9d2a52c0dac WatchSource:0}: Error finding container 18a613528543ce7e29c758afd0250ab8cfd93b48912fa8022a75f9d2a52c0dac: Status 404 returned error can't find the container with id 18a613528543ce7e29c758afd0250ab8cfd93b48912fa8022a75f9d2a52c0dac Jul 06 18:31:03.023343 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:03.023303 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" event={"ID":"66356428-f379-4021-bdf7-dc4ace50dcea","Type":"ContainerStarted","Data":"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066"} Jul 06 18:31:03.023764 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:03.023354 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" event={"ID":"66356428-f379-4021-bdf7-dc4ace50dcea","Type":"ContainerStarted","Data":"18a613528543ce7e29c758afd0250ab8cfd93b48912fa8022a75f9d2a52c0dac"} Jul 06 18:31:03.043428 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:03.043372 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" podStartSLOduration=2.043357756 podStartE2EDuration="2.043357756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:31:03.043078471 +0000 UTC m=+2350.572610847" watchObservedRunningTime="2026-07-06 18:31:03.043357756 +0000 UTC m=+2350.572890126" Jul 06 18:31:05.194178 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:05.194144 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:31:11.908586 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:11.908541 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:31:11.912221 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:11.912193 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:11.912387 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:11.912248 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:11.924844 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:11.924817 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:12.009222 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:12.009190 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:31:12.083693 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:12.083663 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:31:39.811834 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.811799 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:31:39.815386 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.815369 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.817982 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.817959 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"kube-root-ca.crt\"" Jul 06 18:31:39.819140 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.819119 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"rate-test-model-b6a5ee-kserve-self-signed-certs\"" Jul 06 18:31:39.819251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.819161 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"openshift-service-ca.crt\"" Jul 06 18:31:39.819251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.819175 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b6a5ee\"/\"default-dockercfg-7zf4w\"" Jul 06 18:31:39.826939 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.826916 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:31:39.839312 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.839286 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:31:39.961725 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961672 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.961909 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961784 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.961909 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961820 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.961909 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961848 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.961909 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961873 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:39.961909 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:39.961903 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063129 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063166 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063188 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063229 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063222 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063277 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063343 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063661 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063582 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063661 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063603 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.063661 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.063644 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.065516 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.065499 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.065805 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.065789 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.072402 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.072378 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg\") pod \"rate-test-model-b6a5ee-kserve-7fd746c-t5jzn\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.125160 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.125117 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:40.258228 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:40.258203 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:31:40.259862 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:31:40.259838 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaefe8297_ee62_4655_a57f_2ea6af2c8391.slice/crio-a109f9074859951ea8a397e2751ee6f47950aed22b20e1017d957daaabc5e00b WatchSource:0}: Error finding container a109f9074859951ea8a397e2751ee6f47950aed22b20e1017d957daaabc5e00b: Status 404 returned error can't find the container with id a109f9074859951ea8a397e2751ee6f47950aed22b20e1017d957daaabc5e00b Jul 06 18:31:41.238293 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:41.238257 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" event={"ID":"aefe8297-ee62-4655-a57f-2ea6af2c8391","Type":"ContainerStarted","Data":"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3"} Jul 06 18:31:41.238685 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:41.238302 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" event={"ID":"aefe8297-ee62-4655-a57f-2ea6af2c8391","Type":"ContainerStarted","Data":"a109f9074859951ea8a397e2751ee6f47950aed22b20e1017d957daaabc5e00b"} Jul 06 18:31:41.258701 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:41.258644 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" podStartSLOduration=2.2586254 podStartE2EDuration="2.2586254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:31:41.25740856 +0000 UTC m=+2388.786940922" watchObservedRunningTime="2026-07-06 18:31:41.2586254 +0000 UTC m=+2388.788157759" Jul 06 18:31:44.190216 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:44.190174 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:31:50.125343 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:50.125306 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:50.125343 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:50.125352 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:50.137833 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:50.137812 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:50.297454 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:50.297425 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:31:53.164983 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:53.164956 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:31:53.179820 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:53.179796 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:31:57.092916 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:31:57.092880 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:32:11.189241 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:11.189210 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:32:13.572393 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:13.572362 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:13.673049 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:13.673014 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:17.801890 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:17.801807 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:32:17.820669 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:17.820613 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:18.671679 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:18.671649 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:32:18.692852 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:18.692819 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:19.829906 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:19.829852 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:20.185435 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:20.185343 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:20.187106 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:20.187086 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv"] Jul 06 18:32:20.190569 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:20.190544 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-9c9be0-29722710-wqqcv"] Jul 06 18:32:20.250158 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:20.250126 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:21.076560 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:21.076523 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad9f0401-5cc4-454f-9232-ce2768e7e9fb" path="/var/lib/kubelet/pods/ad9f0401-5cc4-454f-9232-ce2768e7e9fb/volumes" Jul 06 18:32:21.476659 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:21.476566 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:21.560307 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:21.560265 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:24.486102 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:24.486064 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:32:25.260483 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:25.260448 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:32:32.826140 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:32.826097 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" podUID="66356428-f379-4021-bdf7-dc4ace50dcea" containerName="main" containerID="cri-o://5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066" gracePeriod=45 Jul 06 18:32:33.069112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.069085 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:32:33.147464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147389 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147422 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147464 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147443 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147688 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147476 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzb9p\" (UniqueName: \"kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147688 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147522 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147688 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147577 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs\") pod \"66356428-f379-4021-bdf7-dc4ace50dcea\" (UID: \"66356428-f379-4021-bdf7-dc4ace50dcea\") " Jul 06 18:32:33.147850 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147741 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache" (OuterVolumeSpecName: "model-cache") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:33.147850 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147705 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home" (OuterVolumeSpecName: "home") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:33.147850 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147809 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:33.148049 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147856 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.148049 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.147874 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.149538 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.149509 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm" (OuterVolumeSpecName: "dshm") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:33.149650 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.149580 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:32:33.149650 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.149579 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p" (OuterVolumeSpecName: "kube-api-access-rzb9p") pod "66356428-f379-4021-bdf7-dc4ace50dcea" (UID: "66356428-f379-4021-bdf7-dc4ace50dcea"). InnerVolumeSpecName "kube-api-access-rzb9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:32:33.248840 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.248808 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.248840 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.248834 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66356428-f379-4021-bdf7-dc4ace50dcea-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.249007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.248848 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66356428-f379-4021-bdf7-dc4ace50dcea-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.249007 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.248860 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzb9p\" (UniqueName: \"kubernetes.io/projected/66356428-f379-4021-bdf7-dc4ace50dcea-kube-api-access-rzb9p\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:33.528731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.528617 2587 generic.go:356] "Generic (PLEG): container finished" podID="66356428-f379-4021-bdf7-dc4ace50dcea" containerID="5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066" exitCode=0 Jul 06 18:32:33.528731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.528665 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" Jul 06 18:32:33.528731 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.528693 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" event={"ID":"66356428-f379-4021-bdf7-dc4ace50dcea","Type":"ContainerDied","Data":"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066"} Jul 06 18:32:33.529001 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.528747 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k" event={"ID":"66356428-f379-4021-bdf7-dc4ace50dcea","Type":"ContainerDied","Data":"18a613528543ce7e29c758afd0250ab8cfd93b48912fa8022a75f9d2a52c0dac"} Jul 06 18:32:33.529001 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.528764 2587 scope.go:117] "RemoveContainer" containerID="5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066" Jul 06 18:32:33.538198 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.538177 2587 scope.go:117] "RemoveContainer" containerID="5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066" Jul 06 18:32:33.538455 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:32:33.538437 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066\": container with ID starting with 5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066 not found: ID does not exist" containerID="5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066" Jul 06 18:32:33.538521 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.538462 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066"} err="failed to get container status \"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066\": rpc error: code = NotFound desc = could not find container \"5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066\": container with ID starting with 5b8d5958416e6b1bc63673c6ab0528b0658c91657eddb001303c8acc21b0b066 not found: ID does not exist" Jul 06 18:32:33.550624 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.550602 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:32:33.556428 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.556409 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9c9be0/rate-test-model-9c9be0-kserve-5f855f7f79-dkb5k"] Jul 06 18:32:33.692486 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.692451 2587 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" podUID="aefe8297-ee62-4655-a57f-2ea6af2c8391" containerName="main" containerID="cri-o://78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3" gracePeriod=45 Jul 06 18:32:33.938588 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:33.938566 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:32:34.054516 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054486 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054523 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054548 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054574 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054612 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054678 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054640 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg\") pod \"aefe8297-ee62-4655-a57f-2ea6af2c8391\" (UID: \"aefe8297-ee62-4655-a57f-2ea6af2c8391\") " Jul 06 18:32:34.054992 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054865 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:34.054992 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054950 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache" (OuterVolumeSpecName: "model-cache") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:34.054992 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.054967 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home" (OuterVolumeSpecName: "home") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:34.056726 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.056690 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm" (OuterVolumeSpecName: "dshm") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 18:32:34.056829 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.056766 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg" (OuterVolumeSpecName: "kube-api-access-gbxrg") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "kube-api-access-gbxrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 18:32:34.056878 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.056824 2587 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aefe8297-ee62-4655-a57f-2ea6af2c8391" (UID: "aefe8297-ee62-4655-a57f-2ea6af2c8391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 18:32:34.155860 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155824 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-tmp-dir\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.155860 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155851 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-dshm\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.156105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155929 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefe8297-ee62-4655-a57f-2ea6af2c8391-tls-certs\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.156105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155940 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-home\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.156105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155951 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefe8297-ee62-4655-a57f-2ea6af2c8391-model-cache\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.156105 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.155960 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxrg\" (UniqueName: \"kubernetes.io/projected/aefe8297-ee62-4655-a57f-2ea6af2c8391-kube-api-access-gbxrg\") on node \"ip-10-0-129-13.ec2.internal\" DevicePath \"\"" Jul 06 18:32:34.536471 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.536383 2587 generic.go:356] "Generic (PLEG): container finished" podID="aefe8297-ee62-4655-a57f-2ea6af2c8391" containerID="78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3" exitCode=0 Jul 06 18:32:34.536471 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.536426 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" Jul 06 18:32:34.536674 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.536463 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" event={"ID":"aefe8297-ee62-4655-a57f-2ea6af2c8391","Type":"ContainerDied","Data":"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3"} Jul 06 18:32:34.536674 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.536500 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn" event={"ID":"aefe8297-ee62-4655-a57f-2ea6af2c8391","Type":"ContainerDied","Data":"a109f9074859951ea8a397e2751ee6f47950aed22b20e1017d957daaabc5e00b"} Jul 06 18:32:34.536674 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.536521 2587 scope.go:117] "RemoveContainer" containerID="78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3" Jul 06 18:32:34.545664 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.545647 2587 scope.go:117] "RemoveContainer" containerID="78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3" Jul 06 18:32:34.545965 ip-10-0-129-13 kubenswrapper[2587]: E0706 18:32:34.545946 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3\": container with ID starting with 78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3 not found: ID does not exist" containerID="78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3" Jul 06 18:32:34.546013 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.545973 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3"} err="failed to get container status \"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3\": rpc error: code = NotFound desc = could not find container \"78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3\": container with ID starting with 78211641df36e5fb90469f55680eae94a26c6ac82c450772bc2bce43db704af3 not found: ID does not exist" Jul 06 18:32:34.556292 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.556266 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:32:34.566397 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:34.564684 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b6a5ee/rate-test-model-b6a5ee-kserve-7fd746c-t5jzn"] Jul 06 18:32:35.076436 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:35.076399 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66356428-f379-4021-bdf7-dc4ace50dcea" path="/var/lib/kubelet/pods/66356428-f379-4021-bdf7-dc4ace50dcea/volumes" Jul 06 18:32:35.076839 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:32:35.076804 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefe8297-ee62-4655-a57f-2ea6af2c8391" path="/var/lib/kubelet/pods/aefe8297-ee62-4655-a57f-2ea6af2c8391/volumes" Jul 06 18:33:02.372137 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:02.372103 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:02.648284 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:02.648187 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:02.902997 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:02.902890 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:03.050473 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:03.050438 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:03.373042 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:03.372993 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:03.381520 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:03.381494 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct"] Jul 06 18:33:03.384057 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:03.384038 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b6a5ee-29722710-gg5ct"] Jul 06 18:33:03.471337 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:03.471309 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:05.076836 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:05.076803 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73426edf-36bb-485f-a945-b53d8c2f79d9" path="/var/lib/kubelet/pods/73426edf-36bb-485f-a945-b53d8c2f79d9/volumes" Jul 06 18:33:05.588755 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:05.588705 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fvp29"] Jul 06 18:33:09.845987 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:09.845948 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:09.904452 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:09.904418 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:10.140893 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:10.140813 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:10.354119 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:10.354078 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:15.838769 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:15.838732 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722695-6ttqz_37a105b8-8714-4e19-9d0f-7f15f6fa5ddb/cleanup/0.log" Jul 06 18:33:15.952146 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:15.952111 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722710-lgnpj_0962c026-223a-4766-a429-f236797f7218/cleanup/0.log" Jul 06 18:33:16.400254 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:16.400228 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8_76f5931f-5426-4cf5-b9a8-698dec958622/copy-manifests/0.log" Jul 06 18:33:16.408396 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:16.408366 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8_76f5931f-5426-4cf5-b9a8-698dec958622/manager/0.log" Jul 06 18:33:17.524969 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.524934 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/util/0.log" Jul 06 18:33:17.531626 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.531605 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/pull/0.log" Jul 06 18:33:17.538270 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.538250 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/extract/0.log" Jul 06 18:33:17.648503 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.648482 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/extract/0.log" Jul 06 18:33:17.654647 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.654624 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/util/0.log" Jul 06 18:33:17.660968 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.660947 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/pull/0.log" Jul 06 18:33:17.774261 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.774227 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/pull/0.log" Jul 06 18:33:17.781322 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.781254 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/extract/0.log" Jul 06 18:33:17.787527 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.787508 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/util/0.log" Jul 06 18:33:17.899584 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.899553 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/util/0.log" Jul 06 18:33:17.906050 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.906020 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/pull/0.log" Jul 06 18:33:17.911993 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:17.911972 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/extract/0.log" Jul 06 18:33:18.377251 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:18.377224 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-n6smm_bdcdd27e-0a04-4efc-96b0-b9b1e07561e1/kuadrant-console-plugin/0.log" Jul 06 18:33:18.735550 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:18.735470 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fvp29_bf663702-0401-480e-8cd8-e7a58513a700/limitador/0.log" Jul 06 18:33:19.552951 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:19.552902 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c4c4d987-mqb8p_2c8cc242-8d7f-4ba8-80df-39ab46aac003/kube-auth-proxy/0.log" Jul 06 18:33:20.713452 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:20.713428 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf6xsl_7a26f793-5ea5-4ca9-ad40-9d6dd02eae77/main/0.log" Jul 06 18:33:24.933614 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.933577 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8v778/must-gather-pgvxk"] Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934004 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefe8297-ee62-4655-a57f-2ea6af2c8391" containerName="main" Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934017 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefe8297-ee62-4655-a57f-2ea6af2c8391" containerName="main" Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934028 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66356428-f379-4021-bdf7-dc4ace50dcea" containerName="main" Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934033 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="66356428-f379-4021-bdf7-dc4ace50dcea" containerName="main" Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934092 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="66356428-f379-4021-bdf7-dc4ace50dcea" containerName="main" Jul 06 18:33:24.934112 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.934102 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefe8297-ee62-4655-a57f-2ea6af2c8391" containerName="main" Jul 06 18:33:24.937471 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.937456 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:24.940273 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.940252 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8v778\"/\"openshift-service-ca.crt\"" Jul 06 18:33:24.941153 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.941137 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8v778\"/\"default-dockercfg-zn8ng\"" Jul 06 18:33:24.941242 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.941160 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8v778\"/\"kube-root-ca.crt\"" Jul 06 18:33:24.945192 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.945171 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8v778/must-gather-pgvxk"] Jul 06 18:33:24.964228 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.964179 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:24.992956 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.992929 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-must-gather-output\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:24.993050 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:24.992975 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm74g\" (UniqueName: \"kubernetes.io/projected/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-kube-api-access-gm74g\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.023525 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.023497 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:25.094072 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.094040 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-must-gather-output\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.094250 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.094185 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm74g\" (UniqueName: \"kubernetes.io/projected/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-kube-api-access-gm74g\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.094404 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.094379 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-must-gather-output\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.105150 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.105125 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm74g\" (UniqueName: \"kubernetes.io/projected/8fe61b8f-df07-41fd-bc3c-728fa64a3f15-kube-api-access-gm74g\") pod \"must-gather-pgvxk\" (UID: \"8fe61b8f-df07-41fd-bc3c-728fa64a3f15\") " pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.247396 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.247317 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8v778/must-gather-pgvxk" Jul 06 18:33:25.385141 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.384994 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8v778/must-gather-pgvxk"] Jul 06 18:33:25.387354 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:33:25.387326 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fe61b8f_df07_41fd_bc3c_728fa64a3f15.slice/crio-f1ba3bd3aa65507d814a33e8ba982ae111e4900c8ee50cf1f60850aee959cfe5 WatchSource:0}: Error finding container f1ba3bd3aa65507d814a33e8ba982ae111e4900c8ee50cf1f60850aee959cfe5: Status 404 returned error can't find the container with id f1ba3bd3aa65507d814a33e8ba982ae111e4900c8ee50cf1f60850aee959cfe5 Jul 06 18:33:25.389419 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.389401 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 18:33:25.809146 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:25.809110 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8v778/must-gather-pgvxk" event={"ID":"8fe61b8f-df07-41fd-bc3c-728fa64a3f15","Type":"ContainerStarted","Data":"f1ba3bd3aa65507d814a33e8ba982ae111e4900c8ee50cf1f60850aee959cfe5"} Jul 06 18:33:26.818934 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:26.818897 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8v778/must-gather-pgvxk" event={"ID":"8fe61b8f-df07-41fd-bc3c-728fa64a3f15","Type":"ContainerStarted","Data":"6fc45f361d318896b81ae34e007c685605f626c41ae4244e666937eaa72dbccc"} Jul 06 18:33:26.819336 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:26.818947 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8v778/must-gather-pgvxk" event={"ID":"8fe61b8f-df07-41fd-bc3c-728fa64a3f15","Type":"ContainerStarted","Data":"1aa617df1bf46ecf1bad5f2e6f2adbf3e5249f3dd6045929b58b250c3686dada"} Jul 06 18:33:26.838165 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:26.838107 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8v778/must-gather-pgvxk" podStartSLOduration=1.9044414760000001 podStartE2EDuration="2.838085074s" totalImagesPullingTime="933.643598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:33:26.835119047 +0000 UTC m=+2494.364651409" watchObservedRunningTime="2026-07-06 18:33:26.838085074 +0000 UTC m=+2494.367617436" Jul 06 18:33:27.962163 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:27.962130 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ms7tt_df8e680c-48eb-41b1-a753-ac4688d2ef96/global-pull-secret-syncer/0.log" Jul 06 18:33:28.059041 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:28.059015 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" Jul 06 18:33:28.138331 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:28.138308 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-13.ec2.internal_ecc2a1fd2e095c365b662a2d934cdfee/haproxy/0.log" Jul 06 18:33:32.948118 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:32.948091 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/extract/0.log" Jul 06 18:33:32.973972 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:32.973945 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/util/0.log" Jul 06 18:33:32.998428 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:32.998392 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v85s8_3c9f302c-fc63-47fb-8ced-0f3f41a97043/pull/0.log" Jul 06 18:33:33.030993 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.030955 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/extract/0.log" Jul 06 18:33:33.057416 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.057392 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/util/0.log" Jul 06 18:33:33.086597 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.086570 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06td7r_552dd939-7828-4a9d-81df-cf0af8dfe045/pull/0.log" Jul 06 18:33:33.120836 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.120804 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/extract/0.log" Jul 06 18:33:33.178928 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.178902 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/util/0.log" Jul 06 18:33:33.221650 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.221539 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sfcmv_3f981780-7213-43f9-97de-4aa3cd23d558/pull/0.log" Jul 06 18:33:33.281779 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.281752 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/extract/0.log" Jul 06 18:33:33.318868 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.318835 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/util/0.log" Jul 06 18:33:33.343812 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.343779 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16npq6_fe79d2de-9450-42e3-ac8c-18e3bd244fd4/pull/0.log" Jul 06 18:33:33.758463 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.758432 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-n6smm_bdcdd27e-0a04-4efc-96b0-b9b1e07561e1/kuadrant-console-plugin/0.log" Jul 06 18:33:33.923489 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:33.923464 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fvp29_bf663702-0401-480e-8cd8-e7a58513a700/limitador/0.log" Jul 06 18:33:34.179633 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:34.179595 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:34.294785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:34.294754 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4crmj_3c190a68-a52e-49b3-b6fc-1bbff1065658/konnectivity-agent/0.log" limit=1024 Jul 06 18:33:35.584682 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.584648 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/alertmanager/0.log" Jul 06 18:33:35.608065 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.608036 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/config-reloader/0.log" Jul 06 18:33:35.629977 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.629951 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/kube-rbac-proxy-web/0.log" Jul 06 18:33:35.650515 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.650483 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/kube-rbac-proxy/0.log" Jul 06 18:33:35.674552 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.674508 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/kube-rbac-proxy-metric/0.log" Jul 06 18:33:35.694897 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.694873 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/prom-label-proxy/0.log" Jul 06 18:33:35.717583 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.717550 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3918fb76-6fb7-49ab-817a-a68283e0298d/init-config-reloader/0.log" Jul 06 18:33:35.895884 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:35.895807 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-7n6np_a9f4da1f-40dd-4690-9373-0eb32200b5c0/monitoring-plugin/0.log" Jul 06 18:33:36.069929 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.069902 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz9rr_b76a5558-6c9e-4b1e-81f6-3e7060417548/node-exporter/0.log" Jul 06 18:33:36.091305 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.091267 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz9rr_b76a5558-6c9e-4b1e-81f6-3e7060417548/kube-rbac-proxy/0.log" Jul 06 18:33:36.118194 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.118160 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz9rr_b76a5558-6c9e-4b1e-81f6-3e7060417548/init-textfile/0.log" Jul 06 18:33:36.391400 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.391365 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-q6nsb_474c0530-039f-4144-a0cc-ff9613a36caf/prometheus-operator/0.log" Jul 06 18:33:36.406669 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.406642 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-q6nsb_474c0530-039f-4144-a0cc-ff9613a36caf/kube-rbac-proxy/0.log" Jul 06 18:33:36.538646 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.538563 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/thanos-query/0.log" Jul 06 18:33:36.560140 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.560101 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/kube-rbac-proxy-web/0.log" Jul 06 18:33:36.584407 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.584378 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/kube-rbac-proxy/0.log" Jul 06 18:33:36.605264 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.605233 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/prom-label-proxy/0.log" Jul 06 18:33:36.627173 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.627145 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/kube-rbac-proxy-rules/0.log" Jul 06 18:33:36.646409 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:36.646363 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb5657dd-p5ls7_ef8f61a7-60bf-4f3c-813c-75b8ac80a75d/kube-rbac-proxy-metrics/0.log" Jul 06 18:33:37.749179 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:37.749144 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-bplpn_8331cd19-bd71-41d3-97df-bf8a2bf42348/networking-console-plugin/0.log" Jul 06 18:33:37.895765 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:37.895690 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77"] Jul 06 18:33:37.902989 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:37.902960 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:37.908474 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:37.908439 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77"] Jul 06 18:33:38.027867 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.027769 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwzk\" (UniqueName: \"kubernetes.io/projected/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-kube-api-access-xjwzk\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.028165 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.028123 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-podres\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.028398 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.028379 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-lib-modules\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.028572 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.028547 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-proc\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.028697 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.028681 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-sys\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.129516 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.129482 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-lib-modules\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.130139 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.130112 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-proc\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.130695 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.130680 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-sys\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.130982 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.130966 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwzk\" (UniqueName: \"kubernetes.io/projected/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-kube-api-access-xjwzk\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.131379 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.131351 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-podres\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.131521 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.131494 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-podres\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.131601 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.129775 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-lib-modules\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.131601 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.130864 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-sys\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.131729 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.130336 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-proc\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.140230 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.140206 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwzk\" (UniqueName: \"kubernetes.io/projected/0d7a1fe3-703f-4c19-b752-6b4f6fdd627b-kube-api-access-xjwzk\") pod \"perf-node-gather-daemonset-z8n77\" (UID: \"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b\") " pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.219226 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.219184 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.409412 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.409376 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77"] Jul 06 18:33:38.411292 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.410991 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/2.log" Jul 06 18:33:38.415616 ip-10-0-129-13 kubenswrapper[2587]: W0706 18:33:38.415590 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d7a1fe3_703f_4c19_b752_6b4f6fdd627b.slice/crio-82d651e966d2f23e12bd941e0a8207c0817165b464ffa2f8673e1cd415ef5a45 WatchSource:0}: Error finding container 82d651e966d2f23e12bd941e0a8207c0817165b464ffa2f8673e1cd415ef5a45: Status 404 returned error can't find the container with id 82d651e966d2f23e12bd941e0a8207c0817165b464ffa2f8673e1cd415ef5a45 Jul 06 18:33:38.427432 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.427401 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lc5tx_d22ec3d0-b287-4110-a439-f96f099270d3/console-operator/3.log" Jul 06 18:33:38.926927 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.926889 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" event={"ID":"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b","Type":"ContainerStarted","Data":"e506d811a5e03165501997bdf1dfa49a6e6e645505a126e4c5f13eaadd6dda76"} Jul 06 18:33:38.927404 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.926951 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" event={"ID":"0d7a1fe3-703f-4c19-b752-6b4f6fdd627b","Type":"ContainerStarted","Data":"82d651e966d2f23e12bd941e0a8207c0817165b464ffa2f8673e1cd415ef5a45"} Jul 06 18:33:38.927404 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.927069 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:38.943942 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:38.943881 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" podStartSLOduration=1.9438591459999999 podStartE2EDuration="1.943859146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:33:38.942560571 +0000 UTC m=+2506.472092950" watchObservedRunningTime="2026-07-06 18:33:38.943859146 +0000 UTC m=+2506.473391505" Jul 06 18:33:39.498109 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:39.498080 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-td6wz_13eea92f-ac58-4933-909b-3fcfcaa50252/volume-data-source-validator/0.log" Jul 06 18:33:40.679840 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:40.679809 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x88hn_19b71253-28c0-43c8-9e13-3f7de5bd5af8/dns/0.log" Jul 06 18:33:40.702726 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:40.702682 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x88hn_19b71253-28c0-43c8-9e13-3f7de5bd5af8/kube-rbac-proxy/0.log" Jul 06 18:33:40.725281 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:40.725259 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lnnp_59cb332e-0f1b-4204-9c67-4df28a5ddbc3/dns-node-resolver/0.log" Jul 06 18:33:41.430770 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:41.430740 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfbkd_efb406b4-8fd5-44d2-a657-6f6691c098a2/node-ca/0.log" Jul 06 18:33:42.534327 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:42.534294 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c4c4d987-mqb8p_2c8cc242-8d7f-4ba8-80df-39ab46aac003/kube-auth-proxy/0.log" Jul 06 18:33:43.377448 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:43.377419 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6j6qd_0fa8514e-3bda-47f6-bed9-f94e0a3682ce/serve-healthcheck-canary/0.log" Jul 06 18:33:43.884853 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:43.884818 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jj9mh_01cb4dd5-3f35-4e37-8b28-a849b62b9e70/insights-operator/0.log" Jul 06 18:33:43.886048 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:43.886024 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jj9mh_01cb4dd5-3f35-4e37-8b28-a849b62b9e70/insights-operator/1.log" Jul 06 18:33:44.032758 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:44.032709 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zcxfd_dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5/kube-rbac-proxy/0.log" Jul 06 18:33:44.053413 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:44.053388 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zcxfd_dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5/exporter/0.log" Jul 06 18:33:44.073849 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:44.073821 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zcxfd_dc91d5d4-5bc5-4abf-abfe-4045ae0ce8a5/extractor/0.log" Jul 06 18:33:44.943837 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:44.943808 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8v778/perf-node-gather-daemonset-z8n77" Jul 06 18:33:46.498578 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:46.498547 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722695-6ttqz_37a105b8-8714-4e19-9d0f-7f15f6fa5ddb/cleanup/0.log" Jul 06 18:33:46.523911 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:46.523886 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722710-lgnpj_0962c026-223a-4766-a429-f236797f7218/cleanup/0.log" Jul 06 18:33:46.743091 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:46.743008 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8_76f5931f-5426-4cf5-b9a8-698dec958622/manager/0.log" Jul 06 18:33:46.763027 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:46.762993 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8c8d8b67-2xgs8_76f5931f-5426-4cf5-b9a8-698dec958622/copy-manifests/0.log" Jul 06 18:33:48.431736 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:48.431680 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-88fc8bffd-kdcmn_b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee/manager/1.log" Jul 06 18:33:48.432450 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:48.432430 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-88fc8bffd-kdcmn_b16e6f66-f5d9-4452-b3d1-c8fb87e7b7ee/manager/0.log" Jul 06 18:33:48.457059 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:48.457033 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-f8sds_f63fbde7-664d-4202-86d4-cfd38098124c/openshift-lws-operator/0.log" Jul 06 18:33:48.793785 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:48.793670 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c4c4d987-mqb8p_2c8cc242-8d7f-4ba8-80df-39ab46aac003/kube-auth-proxy/0.log" Jul 06 18:33:50.237170 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:50.237117 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c4c4d987-mqb8p_2c8cc242-8d7f-4ba8-80df-39ab46aac003/kube-auth-proxy/0.log" Jul 06 18:33:54.774607 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.774571 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2p9dl_6024ad2c-4426-47fd-a007-44e9da76963d/kube-multus/0.log" Jul 06 18:33:54.798673 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.798643 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/kube-multus-additional-cni-plugins/0.log" Jul 06 18:33:54.818425 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.818400 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/egress-router-binary-copy/0.log" Jul 06 18:33:54.840030 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.839944 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/cni-plugins/0.log" Jul 06 18:33:54.861471 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.861445 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/bond-cni-plugin/0.log" Jul 06 18:33:54.881425 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.881401 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/routeoverride-cni/0.log" Jul 06 18:33:54.900046 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.900022 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/whereabouts-cni-bincopy/0.log" Jul 06 18:33:54.919107 ip-10-0-129-13 kubenswrapper[2587]: I0706 18:33:54.919086 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-928jl_d0316b37-42d5-4ac9-86a7-9baecdd16214/whereabouts-cni/0.log"