Jul 15 21:10:04.650946 ip-10-0-139-183 systemd[1]: Starting Kubernetes Kubelet... Jul 15 21:10:05.028434 ip-10-0-139-183 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:10:05.028434 ip-10-0-139-183 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 21:10:05.028434 ip-10-0-139-183 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:10:05.028434 ip-10-0-139-183 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 21:10:05.028434 ip-10-0-139-183 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:10:05.029207 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.029089 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 21:10:05.032489 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032474 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.032489 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032489 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032492 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032496 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032499 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032502 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032504 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032507 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032509 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032512 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032515 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032518 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032520 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032523 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032525 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032528 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032531 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032533 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032536 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032538 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032541 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.032571 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032544 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032546 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032551 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032554 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032558 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032561 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032564 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032567 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032570 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032572 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032575 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032578 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032581 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032584 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032587 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032589 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032592 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032595 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032597 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.033036 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032599 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032602 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032605 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032608 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032610 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032613 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032615 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032617 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032620 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032622 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032625 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032627 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032630 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032632 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032635 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032638 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032641 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032644 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032646 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032649 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.033497 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032652 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032654 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032657 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032659 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032662 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032664 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032667 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032669 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032683 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032686 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032689 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032694 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032698 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032702 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032705 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032710 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032713 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032715 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032718 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.033983 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032721 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032724 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032726 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032729 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032731 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032734 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032736 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032739 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032742 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032744 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032747 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032750 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032752 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032755 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032758 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032760 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032762 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032765 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032767 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.032770 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.034420 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033497 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033504 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033507 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033511 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033514 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033517 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033520 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033523 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033526 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033528 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033531 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033533 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033535 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033538 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033540 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033543 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033546 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033548 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033551 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033554 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.034938 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033556 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033559 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033562 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033564 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033567 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033570 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033572 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033575 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033577 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033580 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033582 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033585 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033587 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033589 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033592 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033594 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033597 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033600 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033605 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033608 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.035442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033610 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033613 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033616 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033619 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033622 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033624 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033629 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033632 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033635 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033638 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033640 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033643 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033645 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033648 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033650 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033653 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033658 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033660 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033663 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033666 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.035916 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033668 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033689 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033693 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033696 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033699 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033702 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033705 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033707 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033710 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033712 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033716 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033720 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033722 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033725 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033727 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033730 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033733 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033736 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033738 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.036399 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033741 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033744 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033746 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033749 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033751 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033754 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033757 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033759 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033762 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033766 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033768 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033771 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033773 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033776 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033778 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033781 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033784 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033786 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033789 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.033793 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.036863 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034149 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034155 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034157 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034161 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034164 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034167 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034169 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034172 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034176 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034180 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034183 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034186 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034189 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034191 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034194 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034196 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034199 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034201 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034205 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.037333 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034208 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034211 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034216 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034219 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034222 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034225 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034228 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034231 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034233 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034236 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034239 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034241 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034244 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034247 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034249 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034252 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034254 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034257 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034260 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034263 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.037797 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034266 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034268 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034271 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034273 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034276 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034279 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034281 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034283 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034286 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034289 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034291 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034293 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034296 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034298 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034303 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034306 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034308 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034311 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034314 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034316 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.038253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034319 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034321 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034324 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034326 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034329 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034332 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034334 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034337 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034339 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034342 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034344 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034347 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034349 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034352 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034354 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034357 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034359 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034362 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034364 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.038732 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034367 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034369 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034372 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034374 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034376 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034380 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034383 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034387 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034389 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034392 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034395 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034397 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034400 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034402 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034406 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034409 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034412 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034414 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034419 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034421 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.034424 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.039181 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034490 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034503 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034509 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034513 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034518 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034521 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034524 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034528 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034532 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034534 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034538 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034541 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034544 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034547 2574 flags.go:64] FLAG: --cgroup-root="" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034549 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034552 2574 flags.go:64] FLAG: --client-ca-file="" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034555 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034558 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034562 2574 flags.go:64] FLAG: --cluster-domain="" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034565 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034568 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034571 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034574 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034577 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 21:10:05.039689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034580 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034583 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034586 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034589 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034592 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034596 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034599 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034602 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034606 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034609 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034611 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034614 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034617 2574 flags.go:64] FLAG: --enable-server="true" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034621 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034624 2574 flags.go:64] FLAG: --event-burst="100" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034627 2574 flags.go:64] FLAG: --event-qps="50" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034630 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034634 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034637 2574 flags.go:64] FLAG: --eviction-hard="" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034640 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034643 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034646 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034648 2574 flags.go:64] FLAG: --eviction-soft="" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034652 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034654 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 21:10:05.040260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034657 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034660 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034663 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034666 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034669 2574 flags.go:64] FLAG: --feature-gates="" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034689 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034694 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034698 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034703 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034707 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034710 2574 flags.go:64] FLAG: --help="false" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034713 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034717 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034720 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034723 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034727 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034730 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034733 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034736 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034738 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034741 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034745 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034748 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034751 2574 flags.go:64] FLAG: --kube-reserved="" Jul 15 21:10:05.040910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034754 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034757 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034760 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034762 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034765 2574 flags.go:64] FLAG: --lock-file="" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034768 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034771 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034774 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034779 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034782 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034785 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034787 2574 flags.go:64] FLAG: --logging-format="text" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034790 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034793 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034796 2574 flags.go:64] FLAG: --manifest-url="" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034799 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034803 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034806 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034809 2574 flags.go:64] FLAG: --max-pods="110" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034812 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034816 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034819 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034822 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034825 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034828 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 21:10:05.041492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034830 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034837 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034840 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034843 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034846 2574 flags.go:64] FLAG: --pod-cidr="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034850 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034855 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034858 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034861 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034864 2574 flags.go:64] FLAG: --port="10250" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034866 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034869 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bdf461384c40f78e" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034872 2574 flags.go:64] FLAG: --qos-reserved="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034875 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034878 2574 flags.go:64] FLAG: --register-node="true" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034880 2574 flags.go:64] FLAG: --register-with-taints="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034884 2574 flags.go:64] FLAG: --registry-burst="10" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034886 2574 flags.go:64] FLAG: --registry-qps="5" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034889 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034892 2574 flags.go:64] FLAG: --reserved-memory="" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034895 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034898 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034901 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034904 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034906 2574 flags.go:64] FLAG: --runonce="false" Jul 15 21:10:05.042136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034909 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034912 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034917 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034919 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034922 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034925 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034928 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034931 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034933 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034936 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034939 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034943 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034946 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034949 2574 flags.go:64] FLAG: --system-cgroups="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034952 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034957 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034960 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034963 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034966 2574 flags.go:64] FLAG: --tls-min-version="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034969 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034972 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034974 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034977 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034980 2574 flags.go:64] FLAG: --v="2" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034984 2574 flags.go:64] FLAG: --version="false" Jul 15 21:10:05.042744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034988 2574 flags.go:64] FLAG: --vmodule="" Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034991 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.034994 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035117 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035121 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035125 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035128 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035131 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035134 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035137 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035142 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035145 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035148 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035150 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035153 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035155 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035158 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035161 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035165 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035167 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.043442 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035170 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035173 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035176 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035179 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035181 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035184 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035186 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035189 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035191 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035194 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035196 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035199 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035201 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035206 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035209 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035212 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035216 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035218 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035221 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.043945 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035223 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035226 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035228 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035231 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035233 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035236 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035239 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035241 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035244 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035246 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035249 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035252 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035255 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035257 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035260 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035263 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035265 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035268 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035270 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.044448 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035273 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035275 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035278 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035280 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035283 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035285 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035288 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035291 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035293 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035295 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035298 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035301 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035303 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035306 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035308 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035312 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035315 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035318 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035321 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035323 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.044921 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035325 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035328 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035330 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035333 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035337 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035339 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035342 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035344 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035347 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035350 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035352 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035354 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035357 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035359 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035362 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035365 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035367 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035369 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035372 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035374 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.045398 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035377 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035379 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035382 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.035384 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.036007 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.042736 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.042753 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.042765 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.042771 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042837 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042843 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042846 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042849 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042852 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.045936 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042856 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042860 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042864 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042866 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042869 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042872 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042877 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042895 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042899 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042902 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042904 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042907 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042910 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042912 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042915 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042918 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042920 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042922 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042925 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042927 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.046289 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042930 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042932 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042935 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042938 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042941 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042944 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042946 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042949 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042952 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042956 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042960 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042964 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042969 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042972 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042975 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042977 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042980 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042982 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042985 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042987 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.046779 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.042997 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043000 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043003 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043005 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043008 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043010 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043013 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043015 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043018 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043021 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043024 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043026 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043029 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043033 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043040 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043047 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043052 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043055 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043058 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.047260 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043060 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043063 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043065 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043068 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043070 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043073 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043075 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043078 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043080 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043082 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043085 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043088 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043090 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043094 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043103 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043105 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043108 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043110 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043114 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043117 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.047722 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043122 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043126 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043130 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043133 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043135 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043138 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043140 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043144 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043146 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043149 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043151 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043153 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043156 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043158 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043161 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.048192 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.043166 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043333 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043338 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043343 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043346 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043349 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043352 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043356 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043361 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043365 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043370 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043374 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043384 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043387 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043390 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043393 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043395 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043398 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043401 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:10:05.048564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043404 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043407 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043410 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043413 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043415 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043418 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043421 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043423 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043425 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043428 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043430 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043433 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043436 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043441 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043445 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043449 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043454 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043457 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043459 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:10:05.049029 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043463 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043467 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043470 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043472 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043476 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043478 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043481 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043490 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043492 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043495 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043498 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043500 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043502 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043505 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043507 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043510 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043512 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043516 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043520 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043524 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:10:05.049481 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043528 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043532 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043534 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043537 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043539 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043542 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043544 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043546 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043549 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043552 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043554 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043557 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043559 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043561 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043564 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043566 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043569 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043571 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043574 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043576 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:10:05.049966 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043585 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043587 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043590 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043592 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043596 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043600 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043604 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043608 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043612 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043614 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043617 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043619 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043622 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043624 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043627 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043629 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043631 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043634 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043636 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043639 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:10:05.050428 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043641 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:05.043644 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.043649 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.043787 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.046795 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.048264 2574 server.go:1019] "Starting client certificate rotation" Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.048374 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.048411 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:10:05.050915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.050616 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:10:05.055317 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.055294 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:10:05.083355 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.083334 2574 log.go:25] "Validated CRI v1 runtime API" Jul 15 21:10:05.085248 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.085236 2574 log.go:25] "Validated CRI v1 image API" Jul 15 21:10:05.086030 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.086016 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 21:10:05.091012 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.090993 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80b9eeaf-7cc5-40db-b03d-c3b0899d7141:/dev/nvme0n1p3 a9db1db0-e660-469e-8211-bc1a2b8ef4a0:/dev/nvme0n1p4] Jul 15 21:10:05.091071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.091012 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 21:10:05.093551 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.093536 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:10:05.096755 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.096634 2574 manager.go:216] Machine: {Timestamp:2026-07-15 21:10:05.09521982 +0000 UTC m=+0.336366319 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094634 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec294b76f05fd170178c334ebdb7e24d SystemUUID:ec294b76-f05f-d170-178c-334ebdb7e24d BootID:37536992-62c1-4be7-9d99-0b2b483f9a21 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cf:0b:33:5e:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cf:0b:33:5e:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:9b:51:48:c1:8d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 21:10:05.096755 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.096751 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 21:10:05.096855 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.096831 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 21:10:05.099934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.099900 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 21:10:05.100063 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.099937 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 21:10:05.100621 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.100610 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 21:10:05.100654 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.100623 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 21:10:05.100654 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.100630 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:10:05.101290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.101279 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:10:05.102474 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.102464 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 21:10:05.105335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105324 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:10:05.105461 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105451 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 21:10:05.105496 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105482 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 15 21:10:05.105496 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105491 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 21:10:05.105548 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105506 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 21:10:05.105548 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105515 2574 kubelet.go:398] "Adding apiserver pod source" Jul 15 21:10:05.105548 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105524 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 21:10:05.105884 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105873 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:10:05.105924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.105890 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:10:05.108041 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.108026 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 21:10:05.109850 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.109835 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 21:10:05.109918 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.109864 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 21:10:05.110031 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110021 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 21:10:05.110064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110036 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 21:10:05.110064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110042 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 21:10:05.110064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110048 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 21:10:05.110064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110056 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 21:10:05.110064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110061 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 21:10:05.110706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110697 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 21:10:05.110737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110707 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 21:10:05.110737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110713 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 21:10:05.110737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110718 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 21:10:05.110737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110726 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 21:10:05.110737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.110735 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 21:10:05.111840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.111830 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 21:10:05.111873 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.111841 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 21:10:05.115391 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.115371 2574 server.go:1268] "Started kubelet" Jul 15 21:10:05.115853 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.115806 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 21:10:05.115923 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.115878 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 21:10:05.115969 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.115939 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 21:10:05.116339 ip-10-0-139-183 systemd[1]: Started Kubernetes Kubelet. Jul 15 21:10:05.117052 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.117027 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 21:10:05.118603 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.118586 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 15 21:10:05.118737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.118713 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:10:05.119264 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.119241 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:10:05.119264 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.119250 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:10:05.122603 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.122580 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 21:10:05.123402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.123389 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 21:10:05.123402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.123394 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 21:10:05.123513 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.123496 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.123989 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.123975 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 21:10:05.123989 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.123987 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 21:10:05.124137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.124002 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 21:10:05.124137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.124098 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 15 21:10:05.124137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.124108 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 15 21:10:05.125624 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.125602 2574 factory.go:55] Registering systemd factory Jul 15 21:10:05.125624 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.125620 2574 factory.go:223] Registration of the systemd container factory successfully Jul 15 21:10:05.126431 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.126419 2574 factory.go:153] Registering CRI-O factory Jul 15 21:10:05.126520 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.126434 2574 factory.go:223] Registration of the crio container factory successfully Jul 15 21:10:05.126520 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.126484 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 21:10:05.126520 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.126507 2574 factory.go:103] Registering Raw factory Jul 15 21:10:05.126520 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.126521 2574 manager.go:1196] Started watching for new ooms in manager Jul 15 21:10:05.127092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.127076 2574 manager.go:349] Starting recovery of all containers Jul 15 21:10:05.128307 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.128284 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7vxdl" Jul 15 21:10:05.133933 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.133907 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 21:10:05.134007 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.133938 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 21:10:05.134541 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.133473 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-183.ec2.internal.18c292cb7ffb659f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-183.ec2.internal,UID:ip-10-0-139-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-183.ec2.internal,},FirstTimestamp:2026-07-15 21:10:05.115336095 +0000 UTC m=+0.356482578,LastTimestamp:2026-07-15 21:10:05.115336095 +0000 UTC m=+0.356482578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-183.ec2.internal,}" Jul 15 21:10:05.136250 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.136226 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7vxdl" Jul 15 21:10:05.136962 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.136948 2574 manager.go:354] Recovery completed Jul 15 21:10:05.142469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.142371 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.144122 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144109 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.144186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144134 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.144186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144144 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.144590 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144578 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.144629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144578 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 21:10:05.144629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144608 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 21:10:05.144629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.144625 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:10:05.145325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145312 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.145372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145335 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.145372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145345 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.145728 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145717 2574 policy_none.go:49] "None policy: Start" Jul 15 21:10:05.145783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145734 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 21:10:05.145783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.145749 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 21:10:05.146372 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.146295 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-183.ec2.internal.18c292cb81b2a1ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-183.ec2.internal,UID:ip-10-0-139-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-183.ec2.internal,},FirstTimestamp:2026-07-15 21:10:05.144121802 +0000 UTC m=+0.385268280,LastTimestamp:2026-07-15 21:10:05.144121802 +0000 UTC m=+0.385268280,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-183.ec2.internal,}" Jul 15 21:10:05.146820 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.146810 2574 policy_none.go:47] "Start" Jul 15 21:10:05.185744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.185728 2574 manager.go:337] "Starting Device Plugin manager" Jul 15 21:10:05.185822 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.185762 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 21:10:05.185822 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.185775 2574 server.go:87] "Starting device plugin registration server" Jul 15 21:10:05.186004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.185992 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 21:10:05.186060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.186007 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 21:10:05.186131 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.186093 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 21:10:05.186205 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.186192 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 21:10:05.186249 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.186206 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 21:10:05.186745 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.186725 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 21:10:05.186824 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.186768 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.285197 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.285124 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 21:10:05.286446 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.286425 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.286446 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.286437 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 21:10:05.286584 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.286478 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 21:10:05.286584 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.286505 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 21:10:05.286584 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.286545 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 21:10:05.287946 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.287928 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.288060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.287964 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.288060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.287981 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.288060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.288013 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.289253 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.289239 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:05.296806 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.296789 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.296879 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.296812 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-183.ec2.internal\": node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.323499 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.323473 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.388581 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.388551 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal"] Jul 15 21:10:05.388666 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.388620 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.390164 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.390143 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.390232 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.390170 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.390232 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.390180 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.391406 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.391393 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.391541 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.391527 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.391610 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.391557 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.392081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392064 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.392081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392072 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.392247 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392091 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.392247 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392097 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.392247 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392107 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.392247 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.392115 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.393251 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.393236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.393314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.393264 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:10:05.393935 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.393918 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:10:05.394021 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.393946 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:10:05.394021 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.393956 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:10:05.421369 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.421351 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-183.ec2.internal\" not found" node="ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.423761 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.423742 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.425857 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.425843 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-183.ec2.internal\" not found" node="ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.426852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.426835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/444dd7fe6c8562d1e4c53584c89f8ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"444dd7fe6c8562d1e4c53584c89f8ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.426903 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.426861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.426903 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.426884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.524801 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.524777 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.528020 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.528115 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.528115 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.528209 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18746a6cabbc4d4e679c4a5e95881be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"18746a6cabbc4d4e679c4a5e95881be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.528209 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/444dd7fe6c8562d1e4c53584c89f8ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"444dd7fe6c8562d1e4c53584c89f8ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.528209 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.528167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/444dd7fe6c8562d1e4c53584c89f8ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"444dd7fe6c8562d1e4c53584c89f8ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.624900 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.624829 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.729473 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.729426 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.729473 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.729466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.729624 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:05.729465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:05.830359 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.830326 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:05.931147 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:05.931084 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.031838 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:06.031810 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.034328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.034310 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:06.047357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.047335 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 21:10:06.047459 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.047441 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 21:10:06.047523 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.047502 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 21:10:06.132776 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:06.132753 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.139888 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.139855 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 21:05:05 +0000 UTC" deadline="2028-04-17 14:43:24.076456108 +0000 UTC" Jul 15 21:10:06.139945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.139888 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15401h33m17.936572556s" Jul 15 21:10:06.233403 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:06.233329 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.334375 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:06.334352 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.434909 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:06.434883 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 15 21:10:06.460699 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.460671 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:06.523671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.523590 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 15 21:10:06.538389 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.538359 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:10:06.538992 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:06.538960 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444dd7fe6c8562d1e4c53584c89f8ea5.slice/crio-925682a0ebd35c4f7b3635cc3500beb199965ca5b3fd41b10e8338b925979bba WatchSource:0}: Error finding container 925682a0ebd35c4f7b3635cc3500beb199965ca5b3fd41b10e8338b925979bba: Status 404 returned error can't find the container with id 925682a0ebd35c4f7b3635cc3500beb199965ca5b3fd41b10e8338b925979bba Jul 15 21:10:06.540202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.540186 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 15 21:10:06.544402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.544381 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:10:06.551185 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.551167 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:10:06.589145 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:06.589116 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:06.708237 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:06.708213 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18746a6cabbc4d4e679c4a5e95881be2.slice/crio-d82ec75284063ce607d8c52b4959ea515f32abe1d11f349dc30d0cd57c0ac76e WatchSource:0}: Error finding container d82ec75284063ce607d8c52b4959ea515f32abe1d11f349dc30d0cd57c0ac76e: Status 404 returned error can't find the container with id d82ec75284063ce607d8c52b4959ea515f32abe1d11f349dc30d0cd57c0ac76e Jul 15 21:10:07.109352 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.109320 2574 apiserver.go:52] "Watching apiserver" Jul 15 21:10:07.120828 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.120802 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 21:10:07.122686 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.122647 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8kgdp","openshift-cluster-node-tuning-operator/tuned-4fcmz","openshift-dns/node-resolver-tb2fz","openshift-image-registry/node-ca-s67rl","openshift-multus/multus-additional-cni-plugins-9p2q6","openshift-multus/multus-k4m4z","openshift-network-diagnostics/network-check-target-nrlr9","openshift-network-operator/iptables-alerter-hq88n","kube-system/global-pull-secret-syncer-vbfsf","kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal","openshift-multus/network-metrics-daemon-6d8qf","openshift-ovn-kubernetes/ovnkube-node-pjmhv"] Jul 15 21:10:07.124540 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.124518 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 21:10:07.125414 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.125385 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:07.125515 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.125483 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:07.127769 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.127736 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.127857 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.127811 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:07.130079 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.130056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.133100 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.133079 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.133199 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.133086 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.133199 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.133111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfs9q\"" Jul 15 21:10:07.133315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.133297 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.135734 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.135702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.136440 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.136241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kvkfc\"" Jul 15 21:10:07.136530 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.136475 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 21:10:07.136586 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.136528 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.136586 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.136529 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:10:07.136695 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.136585 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.138627 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.138173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:07.138627 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.138279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.140816 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.140793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.140956 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.140859 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:07.141435 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.141410 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.141913 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.141893 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 21:10:07.142023 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.141945 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 21:10:07.142096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.142083 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8kck9\"" Jul 15 21:10:07.142368 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.142321 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 21:10:07.142595 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.142577 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.143074 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.142901 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdr5c\"" Jul 15 21:10:07.143273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.143235 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 21:10:07.143273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.143259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.147418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.146333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.147418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.146836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-thzb2\"" Jul 15 21:10:07.147418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.146970 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 21:10:07.147418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.147284 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 21:10:07.149134 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.149068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.149653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.149633 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.149825 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.149806 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.150077 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.149659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8mfzk\"" Jul 15 21:10:07.151905 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.151884 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 21:10:07.152216 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.152193 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 21:10:07.152331 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.152311 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6b9bc\"" Jul 15 21:10:07.152571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.152551 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.153166 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.152778 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.153166 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.152828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.153166 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.153040 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 21:10:07.154328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.154263 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 21:10:07.155045 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.155021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.155887 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.155865 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 21:10:07.156238 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.155991 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ch28\"" Jul 15 21:10:07.156238 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.156026 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.156389 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.156328 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.158955 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.158934 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:10:07.159174 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.159153 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 21:10:07.159259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.159202 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cpldt\"" Jul 15 21:10:07.160878 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.160860 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 21:10:07.169355 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.169336 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tmgrw" Jul 15 21:10:07.176483 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.176461 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tmgrw" Jul 15 21:10:07.225399 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.225377 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 21:10:07.232987 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.232965 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:07.238737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-kubelet-config\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.238840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.238840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-tuned\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.238840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-multus-certs\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.238840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jgm\" (UniqueName: \"kubernetes.io/projected/ce512db5-dc5d-43ff-acd1-5e7f20b33068-kube-api-access-r9jgm\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-agent-certs\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-bin\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-script-lib\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9r6w\" (UniqueName: \"kubernetes.io/projected/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-kube-api-access-d9r6w\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.238958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-hostroot\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.239061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-var-lib-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-system-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-kubelet\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8622b0a-797d-4b91-917a-ea4f58d59733-ovn-node-metrics-cert\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74n6\" (UniqueName: \"kubernetes.io/projected/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-kube-api-access-k74n6\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-bin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-etc-kubernetes\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvb8\" (UniqueName: \"kubernetes.io/projected/39348237-6557-4cce-bea5-eb23ba277a4a-kube-api-access-vkvb8\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:07.239415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxmg\" (UniqueName: \"kubernetes.io/projected/087d35b7-30e7-4401-a10a-d5d062b290ce-kube-api-access-7zxmg\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-systemd-units\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed301111-8cdd-42f1-b985-2dcac8b2d466-iptables-alerter-script\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-tmp-dir\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-multus-daemon-config\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-run\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-kubelet\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-log-socket\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysconfig\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-slash\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.239953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.239920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hcm\" (UniqueName: \"kubernetes.io/projected/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-kube-api-access-w7hcm\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-cnibin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-kubernetes\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-konnectivity-ca\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-socket-dir-parent\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.240576 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-sys\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-host\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-etc-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-netd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-cni-binary-copy\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-dbus\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-device-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-ovn\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjfm\" (UniqueName: \"kubernetes.io/projected/e8622b0a-797d-4b91-917a-ea4f58d59733-kube-api-access-lmjfm\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-os-release\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-k8s-cni-cncf-io\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-conf\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.240945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cnibin\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.240980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-modprobe-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-lib-modules\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-netns\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-multus\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-conf-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-system-cni-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-systemd\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-tmp\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-os-release\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xzv\" (UniqueName: \"kubernetes.io/projected/c8701063-1150-4ed0-b93d-df0fec133df0-kube-api-access-c8xzv\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.241671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce512db5-dc5d-43ff-acd1-5e7f20b33068-serviceca\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-var-lib-kubelet\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-netns\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-config\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed301111-8cdd-42f1-b985-2dcac8b2d466-host-slash\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-hosts-file\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce512db5-dc5d-43ff-acd1-5e7f20b33068-host\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-systemd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-node-log\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-env-overrides\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.242522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.241618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5t8\" (UniqueName: \"kubernetes.io/projected/ed301111-8cdd-42f1-b985-2dcac8b2d466-kube-api-access-4n5t8\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.249226 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.249201 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:07.249340 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.249237 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:07.249340 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.249252 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:07.249449 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.249343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:07.749300894 +0000 UTC m=+2.990447376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:07.294052 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.294001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"18746a6cabbc4d4e679c4a5e95881be2","Type":"ContainerStarted","Data":"d82ec75284063ce607d8c52b4959ea515f32abe1d11f349dc30d0cd57c0ac76e"} Jul 15 21:10:07.296021 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.295965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" event={"ID":"444dd7fe6c8562d1e4c53584c89f8ea5","Type":"ContainerStarted","Data":"925682a0ebd35c4f7b3635cc3500beb199965ca5b3fd41b10e8338b925979bba"} Jul 15 21:10:07.342796 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-ovn\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.342925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjfm\" (UniqueName: \"kubernetes.io/projected/e8622b0a-797d-4b91-917a-ea4f58d59733-kube-api-access-lmjfm\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.342925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-ovn\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.342925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-os-release\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-k8s-cni-cncf-io\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.342999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-conf\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cnibin\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-modprobe-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.343158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-lib-modules\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-os-release\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-netns\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-multus\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-k8s-cni-cncf-io\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-conf-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-system-cni-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-systemd\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.343335 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-tmp\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.343394 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:07.843374431 +0000 UTC m=+3.084520913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-os-release\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xzv\" (UniqueName: \"kubernetes.io/projected/c8701063-1150-4ed0-b93d-df0fec133df0-kube-api-access-c8xzv\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce512db5-dc5d-43ff-acd1-5e7f20b33068-serviceca\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.343611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-var-lib-kubelet\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-netns\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-config\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed301111-8cdd-42f1-b985-2dcac8b2d466-host-slash\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-hosts-file\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce512db5-dc5d-43ff-acd1-5e7f20b33068-host\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-systemd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-node-log\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-env-overrides\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-os-release\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5t8\" (UniqueName: \"kubernetes.io/projected/ed301111-8cdd-42f1-b985-2dcac8b2d466-kube-api-access-4n5t8\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.343809 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-kubelet-config\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.343864 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:07.843844084 +0000 UTC m=+3.084990564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-kubelet-config\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.344493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-tuned\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.343987 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-var-lib-kubelet\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-multus-certs\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jgm\" (UniqueName: \"kubernetes.io/projected/ce512db5-dc5d-43ff-acd1-5e7f20b33068-kube-api-access-r9jgm\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-agent-certs\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce512db5-dc5d-43ff-acd1-5e7f20b33068-serviceca\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-hosts-file\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-bin\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-run-netns\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-system-cni-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344203 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-script-lib\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-systemd\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344236 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-bin\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9r6w\" (UniqueName: \"kubernetes.io/projected/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-kube-api-access-d9r6w\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.345433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-hostroot\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-var-lib-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-multus\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-conf\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-conf-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-system-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cnibin\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-kubelet\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8622b0a-797d-4b91-917a-ea4f58d59733-ovn-node-metrics-cert\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74n6\" (UniqueName: \"kubernetes.io/projected/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-kube-api-access-k74n6\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-bin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344102 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-netns\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-node-log\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-cni-bin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce512db5-dc5d-43ff-acd1-5e7f20b33068-host\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-config\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.346362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-systemd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed301111-8cdd-42f1-b985-2dcac8b2d466-host-slash\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-etc-kubernetes\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvb8\" (UniqueName: \"kubernetes.io/projected/39348237-6557-4cce-bea5-eb23ba277a4a-kube-api-access-vkvb8\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxmg\" (UniqueName: \"kubernetes.io/projected/087d35b7-30e7-4401-a10a-d5d062b290ce-kube-api-access-7zxmg\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-systemd-units\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed301111-8cdd-42f1-b985-2dcac8b2d466-iptables-alerter-script\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-tmp-dir\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-multus-daemon-config\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-run\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-ovnkube-script-lib\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-kubelet\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.344134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.347290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-run-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-system-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-etc-kubernetes\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-log-socket\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysconfig\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-slash\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hcm\" (UniqueName: \"kubernetes.io/projected/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-kube-api-access-w7hcm\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-cnibin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-kubernetes\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-konnectivity-ca\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-socket-dir-parent\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-sys\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.348402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-sys\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-host\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-etc-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-netd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-cni-binary-copy\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysctl-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-dbus\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-device-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345592 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-hostroot\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345938 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed301111-8cdd-42f1-b985-2dcac8b2d466-iptables-alerter-script\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.345980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-kubernetes\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-socket-dir-parent\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-konnectivity-ca\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-var-lib-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-host\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-etc-openvswitch\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-cni-netd\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.349179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-tmp-dir\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.346886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-var-lib-kubelet\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-cni-binary-copy\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-log-socket\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-run\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-kubelet\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-systemd-units\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8622b0a-797d-4b91-917a-ea4f58d59733-host-slash\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64381618-e5ba-4a2d-aeeb-c435bffe9ead-dbus\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39348237-6557-4cce-bea5-eb23ba277a4a-device-dir\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-sysconfig\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8701063-1150-4ed0-b93d-df0fec133df0-multus-daemon-config\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-lib-modules\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-modprobe-d\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-cnibin\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-host-run-multus-certs\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.347951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8701063-1150-4ed0-b93d-df0fec133df0-multus-cni-dir\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.350092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.348209 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.351058 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.348303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8622b0a-797d-4b91-917a-ea4f58d59733-ovn-node-metrics-cert\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.351058 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.348392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcb762e6-ed5a-4972-b65e-97ac73fad4fa-agent-certs\") pod \"konnectivity-agent-8kgdp\" (UID: \"dcb762e6-ed5a-4972-b65e-97ac73fad4fa\") " pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.351058 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.348611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8622b0a-797d-4b91-917a-ea4f58d59733-env-overrides\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.351058 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.349873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-etc-tuned\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.351551 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.351529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087d35b7-30e7-4401-a10a-d5d062b290ce-tmp\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.357239 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.357183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjfm\" (UniqueName: \"kubernetes.io/projected/e8622b0a-797d-4b91-917a-ea4f58d59733-kube-api-access-lmjfm\") pod \"ovnkube-node-pjmhv\" (UID: \"e8622b0a-797d-4b91-917a-ea4f58d59733\") " pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.362191 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.362137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9r6w\" (UniqueName: \"kubernetes.io/projected/9469fb72-dc7b-438a-8d69-5c0fc6aff7da-kube-api-access-d9r6w\") pod \"node-resolver-tb2fz\" (UID: \"9469fb72-dc7b-438a-8d69-5c0fc6aff7da\") " pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.362691 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.362671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jgm\" (UniqueName: \"kubernetes.io/projected/ce512db5-dc5d-43ff-acd1-5e7f20b33068-kube-api-access-r9jgm\") pod \"node-ca-s67rl\" (UID: \"ce512db5-dc5d-43ff-acd1-5e7f20b33068\") " pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.363157 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.363137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxmg\" (UniqueName: \"kubernetes.io/projected/087d35b7-30e7-4401-a10a-d5d062b290ce-kube-api-access-7zxmg\") pod \"tuned-4fcmz\" (UID: \"087d35b7-30e7-4401-a10a-d5d062b290ce\") " pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.363606 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.363500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74n6\" (UniqueName: \"kubernetes.io/projected/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-kube-api-access-k74n6\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.363702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.363615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvb8\" (UniqueName: \"kubernetes.io/projected/39348237-6557-4cce-bea5-eb23ba277a4a-kube-api-access-vkvb8\") pod \"aws-ebs-csi-driver-node-bkvv9\" (UID: \"39348237-6557-4cce-bea5-eb23ba277a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.364074 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.364018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xzv\" (UniqueName: \"kubernetes.io/projected/c8701063-1150-4ed0-b93d-df0fec133df0-kube-api-access-c8xzv\") pod \"multus-k4m4z\" (UID: \"c8701063-1150-4ed0-b93d-df0fec133df0\") " pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.364417 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.364373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5t8\" (UniqueName: \"kubernetes.io/projected/ed301111-8cdd-42f1-b985-2dcac8b2d466-kube-api-access-4n5t8\") pod \"iptables-alerter-hq88n\" (UID: \"ed301111-8cdd-42f1-b985-2dcac8b2d466\") " pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.365554 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.365512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hcm\" (UniqueName: \"kubernetes.io/projected/fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87-kube-api-access-w7hcm\") pod \"multus-additional-cni-plugins-9p2q6\" (UID: \"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87\") " pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.444581 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.444547 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tb2fz" Jul 15 21:10:07.455219 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.455193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s67rl" Jul 15 21:10:07.461800 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.461773 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9469fb72_dc7b_438a_8d69_5c0fc6aff7da.slice/crio-5a968e1e990cf97697e31ddc61e2bde729c36eebc5b91fd111ea71b96c96f1d4 WatchSource:0}: Error finding container 5a968e1e990cf97697e31ddc61e2bde729c36eebc5b91fd111ea71b96c96f1d4: Status 404 returned error can't find the container with id 5a968e1e990cf97697e31ddc61e2bde729c36eebc5b91fd111ea71b96c96f1d4 Jul 15 21:10:07.466538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.466358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" Jul 15 21:10:07.468369 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.468343 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce512db5_dc5d_43ff_acd1_5e7f20b33068.slice/crio-5e8104153d05cdb8a62efebfa26db7544faa7b06e8996caa3cc11520d79103f8 WatchSource:0}: Error finding container 5e8104153d05cdb8a62efebfa26db7544faa7b06e8996caa3cc11520d79103f8: Status 404 returned error can't find the container with id 5e8104153d05cdb8a62efebfa26db7544faa7b06e8996caa3cc11520d79103f8 Jul 15 21:10:07.473490 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.473470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k4m4z" Jul 15 21:10:07.480239 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.480210 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2f32a9_f96f_4c0b_9480_1d54dc1e7b87.slice/crio-74dec62cf7b2ad6fbf9fe8c9765e9e53af40ec9924b324d15729bb3ba2bf0f14 WatchSource:0}: Error finding container 74dec62cf7b2ad6fbf9fe8c9765e9e53af40ec9924b324d15729bb3ba2bf0f14: Status 404 returned error can't find the container with id 74dec62cf7b2ad6fbf9fe8c9765e9e53af40ec9924b324d15729bb3ba2bf0f14 Jul 15 21:10:07.481899 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.481783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:07.485727 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.485661 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8701063_1150_4ed0_b93d_df0fec133df0.slice/crio-efb6559f60d59a2f04ab8ff9cde58e7a91a515d80282ae2dbfd0acbe14999d3b WatchSource:0}: Error finding container efb6559f60d59a2f04ab8ff9cde58e7a91a515d80282ae2dbfd0acbe14999d3b: Status 404 returned error can't find the container with id efb6559f60d59a2f04ab8ff9cde58e7a91a515d80282ae2dbfd0acbe14999d3b Jul 15 21:10:07.491297 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.491277 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" Jul 15 21:10:07.491963 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.491939 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb762e6_ed5a_4972_b65e_97ac73fad4fa.slice/crio-2b7e8960d0c445c7bf8da337d476b1cd475f47db660bd6281444e987c7e3b55e WatchSource:0}: Error finding container 2b7e8960d0c445c7bf8da337d476b1cd475f47db660bd6281444e987c7e3b55e: Status 404 returned error can't find the container with id 2b7e8960d0c445c7bf8da337d476b1cd475f47db660bd6281444e987c7e3b55e Jul 15 21:10:07.497917 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.497770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:07.502871 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.502848 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087d35b7_30e7_4401_a10a_d5d062b290ce.slice/crio-3c169d404d3a2c64d8373477b735de952db7ce3ad91d6aa3d58855bdedf763e6 WatchSource:0}: Error finding container 3c169d404d3a2c64d8373477b735de952db7ce3ad91d6aa3d58855bdedf763e6: Status 404 returned error can't find the container with id 3c169d404d3a2c64d8373477b735de952db7ce3ad91d6aa3d58855bdedf763e6 Jul 15 21:10:07.507943 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.507748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" Jul 15 21:10:07.511978 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.511953 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8622b0a_797d_4b91_917a_ea4f58d59733.slice/crio-dbc0b83dbad13d707b5c687b59361ec9b5ff1dfcab5ecb224f72b9a7c85e3969 WatchSource:0}: Error finding container dbc0b83dbad13d707b5c687b59361ec9b5ff1dfcab5ecb224f72b9a7c85e3969: Status 404 returned error can't find the container with id dbc0b83dbad13d707b5c687b59361ec9b5ff1dfcab5ecb224f72b9a7c85e3969 Jul 15 21:10:07.514303 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.514089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq88n" Jul 15 21:10:07.519987 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.519963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39348237_6557_4cce_bea5_eb23ba277a4a.slice/crio-1f3d5bc521a9169293634c032c87afaebba6cc61629e26d53b6be5a0ac4c574a WatchSource:0}: Error finding container 1f3d5bc521a9169293634c032c87afaebba6cc61629e26d53b6be5a0ac4c574a: Status 404 returned error can't find the container with id 1f3d5bc521a9169293634c032c87afaebba6cc61629e26d53b6be5a0ac4c574a Jul 15 21:10:07.525127 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:10:07.525106 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded301111_8cdd_42f1_b985_2dcac8b2d466.slice/crio-f0dd04f2764a62f2277fd32ea81589c7306343bb010725753a18d13013b7c608 WatchSource:0}: Error finding container f0dd04f2764a62f2277fd32ea81589c7306343bb010725753a18d13013b7c608: Status 404 returned error can't find the container with id f0dd04f2764a62f2277fd32ea81589c7306343bb010725753a18d13013b7c608 Jul 15 21:10:07.670159 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.670077 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:10:07.750072 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.750037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:07.750283 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.750261 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:07.750361 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.750295 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:07.750361 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.750309 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:07.750507 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.750381 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:08.750361617 +0000 UTC m=+3.991508088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:07.851293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.851262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:07.851492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:07.851306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:07.851492 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.851430 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:07.851492 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.851465 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:07.851659 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.851503 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:08.851487228 +0000 UTC m=+4.092633694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:07.851659 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:07.851520 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:08.851512442 +0000 UTC m=+4.092658907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:08.179992 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.179920 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:05:07 +0000 UTC" deadline="2028-03-25 00:02:12.005647094 +0000 UTC" Jul 15 21:10:08.179992 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.179982 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14834h52m3.825669063s" Jul 15 21:10:08.286870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.286826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:08.286870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.286877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:08.287125 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.287012 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:08.287412 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.287311 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:08.300613 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.300472 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"dbc0b83dbad13d707b5c687b59361ec9b5ff1dfcab5ecb224f72b9a7c85e3969"} Jul 15 21:10:08.302270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.302173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8kgdp" event={"ID":"dcb762e6-ed5a-4972-b65e-97ac73fad4fa","Type":"ContainerStarted","Data":"2b7e8960d0c445c7bf8da337d476b1cd475f47db660bd6281444e987c7e3b55e"} Jul 15 21:10:08.303713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.303666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerStarted","Data":"74dec62cf7b2ad6fbf9fe8c9765e9e53af40ec9924b324d15729bb3ba2bf0f14"} Jul 15 21:10:08.305284 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.305236 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s67rl" event={"ID":"ce512db5-dc5d-43ff-acd1-5e7f20b33068","Type":"ContainerStarted","Data":"5e8104153d05cdb8a62efebfa26db7544faa7b06e8996caa3cc11520d79103f8"} Jul 15 21:10:08.306596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.306570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tb2fz" event={"ID":"9469fb72-dc7b-438a-8d69-5c0fc6aff7da","Type":"ContainerStarted","Data":"5a968e1e990cf97697e31ddc61e2bde729c36eebc5b91fd111ea71b96c96f1d4"} Jul 15 21:10:08.308394 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.308342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq88n" event={"ID":"ed301111-8cdd-42f1-b985-2dcac8b2d466","Type":"ContainerStarted","Data":"f0dd04f2764a62f2277fd32ea81589c7306343bb010725753a18d13013b7c608"} Jul 15 21:10:08.310103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.310076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" event={"ID":"39348237-6557-4cce-bea5-eb23ba277a4a","Type":"ContainerStarted","Data":"1f3d5bc521a9169293634c032c87afaebba6cc61629e26d53b6be5a0ac4c574a"} Jul 15 21:10:08.312291 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.312235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" event={"ID":"087d35b7-30e7-4401-a10a-d5d062b290ce","Type":"ContainerStarted","Data":"3c169d404d3a2c64d8373477b735de952db7ce3ad91d6aa3d58855bdedf763e6"} Jul 15 21:10:08.313618 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.313592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k4m4z" event={"ID":"c8701063-1150-4ed0-b93d-df0fec133df0","Type":"ContainerStarted","Data":"efb6559f60d59a2f04ab8ff9cde58e7a91a515d80282ae2dbfd0acbe14999d3b"} Jul 15 21:10:08.761035 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.760296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:08.761035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.760532 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:08.761035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.760555 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:08.761035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.760569 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:08.761035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.760641 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:10.760620323 +0000 UTC m=+6.001766795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:08.861740 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.861479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:08.861740 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:08.861586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:08.861740 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.861671 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:08.862035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.861801 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:10.861774361 +0000 UTC m=+6.102920849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:08.862103 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.861702 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:08.862163 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:08.862155 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:10.862127624 +0000 UTC m=+6.103274095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:09.180620 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:09.180578 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:05:07 +0000 UTC" deadline="2028-03-24 19:19:16.458655438 +0000 UTC" Jul 15 21:10:09.180620 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:09.180618 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14830h9m7.278043223s" Jul 15 21:10:09.289691 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:09.289659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:09.289903 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:09.289825 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:10.287289 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:10.287255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:10.287794 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.287393 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:10.287898 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:10.287808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:10.287961 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.287919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:10.779005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:10.778899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:10.779169 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.779134 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:10.779169 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.779157 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:10.779292 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.779171 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:10.779292 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.779234 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:14.77921286 +0000 UTC m=+10.020359340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:10.880386 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:10.880104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:10.880386 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:10.880152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:10.880386 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.880286 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:10.880386 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.880315 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:10.880386 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.880382 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:14.880360102 +0000 UTC m=+10.121506572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:10.880837 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:10.880405 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:14.880393398 +0000 UTC m=+10.121539868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:11.288835 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:11.287496 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:11.288835 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:11.287638 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:12.287643 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:12.287607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:12.287911 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:12.287654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:12.287911 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:12.287770 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:12.288045 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:12.287916 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:13.290997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:13.290695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:13.291420 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:13.291121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:14.287158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:14.287120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:14.287158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:14.287151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:14.287420 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.287276 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:14.287420 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.287362 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:14.816328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:14.816257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:14.816909 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.816414 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:14.816909 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.816436 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:14.816909 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.816449 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:14.816909 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.816508 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:22.816489087 +0000 UTC m=+18.057635564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:14.917114 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:14.917074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:14.917114 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:14.917122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:14.917611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.917333 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:14.917611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.917404 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:22.917383944 +0000 UTC m=+18.158530412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:14.917611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.917412 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:14.917611 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:14.917448 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:22.917436777 +0000 UTC m=+18.158583249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:15.288662 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:15.288159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:15.288662 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:15.288271 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:16.287615 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:16.286891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:16.287615 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:16.287034 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:16.287615 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:16.287459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:16.287615 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:16.287567 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:17.287694 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:17.287653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:17.288130 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:17.287811 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:18.287311 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:18.287277 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:18.287486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:18.287286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:18.287486 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:18.287399 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:18.287486 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:18.287471 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:19.287610 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:19.287572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:19.288058 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:19.287708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:20.287686 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:20.287640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:20.288119 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:20.287645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:20.288119 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:20.287781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:20.288119 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:20.287855 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:21.287201 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:21.287166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:21.287356 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:21.287294 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:22.286823 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:22.286785 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:22.287274 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:22.286785 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:22.287274 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.286916 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:22.287274 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.286991 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:22.872684 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:22.872642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:22.872838 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.872769 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:22.872838 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.872785 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:22.872838 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.872797 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:22.872951 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.872860 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:38.872844026 +0000 UTC m=+34.113990504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:22.973878 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:22.973839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:22.974035 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:22.973895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:22.974035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.973977 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:22.974112 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.974043 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:22.974112 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.974045 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:10:38.974029629 +0000 UTC m=+34.215176094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:22.974112 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:22.974107 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:38.974089127 +0000 UTC m=+34.215235592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:23.287386 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:23.287308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:23.287800 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:23.287426 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:24.287113 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:24.287077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:24.287293 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:24.287219 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:24.287293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:24.287275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:24.287421 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:24.287396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:25.288130 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:25.288066 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:25.288578 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:25.288182 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:26.287502 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:26.287470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:26.287700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:26.287514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:26.287700 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:26.287599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:26.287828 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:26.287767 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:27.286990 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:27.286954 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:27.287420 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:27.287081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:28.287578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.287283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:28.288572 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.287323 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:28.288572 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:28.287648 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:28.288572 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:28.287808 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:28.372946 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.372912 2574 generic.go:356] "Generic (PLEG): container finished" podID="18746a6cabbc4d4e679c4a5e95881be2" containerID="ef711aaf671b640a91365d1fcc44daccc937e1cefd3ed8e80c30bf5ef95e0791" exitCode=0 Jul 15 21:10:28.373052 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.372993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"18746a6cabbc4d4e679c4a5e95881be2","Type":"ContainerDied","Data":"ef711aaf671b640a91365d1fcc44daccc937e1cefd3ed8e80c30bf5ef95e0791"} Jul 15 21:10:28.376133 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:10:28.376504 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376475 2574 generic.go:356] "Generic (PLEG): container finished" podID="e8622b0a-797d-4b91-917a-ea4f58d59733" containerID="a6f8497dd27ddb227abf9d076c2059ca1bff1214327f9e9d63d2960cc095f40a" exitCode=1 Jul 15 21:10:28.376609 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"e6ae026937f0ab145fbb68de6af0c32d7135535734700de037aa967ac0451a36"} Jul 15 21:10:28.376696 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"3f10716bcc860df3a4d9562cd5842b06b997602e9b6a8d66eca3ab14a8638344"} Jul 15 21:10:28.376696 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376693 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"e335782fc7a343cac853ecfd01653aa40e6178507cf42c452e115481a060b6a9"} Jul 15 21:10:28.376804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"4e4d1c108bb99f5ab127cd5d6199d87d6ddc5b6075452550144716686a996d08"} Jul 15 21:10:28.376804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerDied","Data":"a6f8497dd27ddb227abf9d076c2059ca1bff1214327f9e9d63d2960cc095f40a"} Jul 15 21:10:28.376892 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.376834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"32a5561b824e9a91feea598ce7a2b0e3702d68ce6326b906865ada98775ce161"} Jul 15 21:10:28.379500 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.379478 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="118c6359bdd515d0aa8824cab9e51ec76ab85d5ac3afc16b98c42cbce9f603de" exitCode=0 Jul 15 21:10:28.379569 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.379554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"118c6359bdd515d0aa8824cab9e51ec76ab85d5ac3afc16b98c42cbce9f603de"} Jul 15 21:10:28.386731 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.386705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s67rl" event={"ID":"ce512db5-dc5d-43ff-acd1-5e7f20b33068","Type":"ContainerStarted","Data":"1824e2ec00ba2a83fb59712b94f952e778af55e8d6efcd1a249d4ab15673fca2"} Jul 15 21:10:28.388090 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.388061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tb2fz" event={"ID":"9469fb72-dc7b-438a-8d69-5c0fc6aff7da","Type":"ContainerStarted","Data":"23dc1b02d79094610e88ecc47e653a5a7c3f071cd93bb8edc821d74110f8bf39"} Jul 15 21:10:28.389381 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.389359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" event={"ID":"444dd7fe6c8562d1e4c53584c89f8ea5","Type":"ContainerStarted","Data":"f42cb8fb4baa97c716e15d2b30f0c909b2b8823f3894d1bb57b0b9cf9d5de9d5"} Jul 15 21:10:28.390654 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.390629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" event={"ID":"39348237-6557-4cce-bea5-eb23ba277a4a","Type":"ContainerStarted","Data":"6c267c69411a81eee7b000354482710fc72f9d395242f21a732cf8518b5972dd"} Jul 15 21:10:28.391895 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.391877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" event={"ID":"087d35b7-30e7-4401-a10a-d5d062b290ce","Type":"ContainerStarted","Data":"ed8ec6877d8fbab73f176fc212148ed31c6c01e1dcda79ad9dc939ee12c220f0"} Jul 15 21:10:28.393066 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.393047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k4m4z" event={"ID":"c8701063-1150-4ed0-b93d-df0fec133df0","Type":"ContainerStarted","Data":"57f0a9ea77bdfb338cd82b022e7bf358e4a522ffd0b18eb64379c8bdf27ba769"} Jul 15 21:10:28.474133 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.474072 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s67rl" podStartSLOduration=2.497914955 podStartE2EDuration="22.474049128s" totalImagesPullingTime="19.976134173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:28.437011668 +0000 UTC m=+23.678158165" watchObservedRunningTime="2026-07-15 21:10:28.474049128 +0000 UTC m=+23.715195615" Jul 15 21:10:28.537605 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.537563 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tb2fz" podStartSLOduration=2.5721713729999998 podStartE2EDuration="22.53754806s" totalImagesPullingTime="19.965376687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:28.497365991 +0000 UTC m=+23.738512481" watchObservedRunningTime="2026-07-15 21:10:28.53754806 +0000 UTC m=+23.778694547" Jul 15 21:10:28.573506 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.573432 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" podStartSLOduration=21.573418045 podStartE2EDuration="21.573418045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:28.538146818 +0000 UTC m=+23.779293304" watchObservedRunningTime="2026-07-15 21:10:28.573418045 +0000 UTC m=+23.814564555" Jul 15 21:10:28.573973 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.573945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k4m4z" podStartSLOduration=2.040681331 podStartE2EDuration="22.573935452s" totalImagesPullingTime="20.533254121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:28.573186328 +0000 UTC m=+23.814332858" watchObservedRunningTime="2026-07-15 21:10:28.573935452 +0000 UTC m=+23.815081939" Jul 15 21:10:28.598565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:28.598526 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4fcmz" podStartSLOduration=2.657129436 podStartE2EDuration="22.598515616s" totalImagesPullingTime="19.94138618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:28.598229701 +0000 UTC m=+23.839376187" watchObservedRunningTime="2026-07-15 21:10:28.598515616 +0000 UTC m=+23.839662102" Jul 15 21:10:29.287667 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.287636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:29.288213 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:29.287751 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:29.396544 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.396516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq88n" event={"ID":"ed301111-8cdd-42f1-b985-2dcac8b2d466","Type":"ContainerStarted","Data":"716acaa7d5ecc6e2d77923b6d44ac099175604b957cf02cf4aed6dbffc3f8b4e"} Jul 15 21:10:29.397979 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.397960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"18746a6cabbc4d4e679c4a5e95881be2","Type":"ContainerStarted","Data":"4ab8491a7df581c72fd2961cd4b1dc53a0b21b1196a0cc300e4b904708a2e64e"} Jul 15 21:10:29.399254 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.399231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8kgdp" event={"ID":"dcb762e6-ed5a-4972-b65e-97ac73fad4fa","Type":"ContainerStarted","Data":"3f52a3d98ca3fcbb4c6b36b352f6b3c81c5b48ed17634d302633b9df3a8a0344"} Jul 15 21:10:29.416322 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.416284 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hq88n" podStartSLOduration=3.499029536 podStartE2EDuration="23.416272376s" totalImagesPullingTime="19.91724284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:29.415915492 +0000 UTC m=+24.657062001" watchObservedRunningTime="2026-07-15 21:10:29.416272376 +0000 UTC m=+24.657418863" Jul 15 21:10:29.434002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.433956 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8kgdp" podStartSLOduration=3.483789028 podStartE2EDuration="23.433939763s" totalImagesPullingTime="19.950150735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:29.43351601 +0000 UTC m=+24.674662498" watchObservedRunningTime="2026-07-15 21:10:29.433939763 +0000 UTC m=+24.675086251" Jul 15 21:10:29.438288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.438271 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 21:10:29.450824 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.450789 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" podStartSLOduration=22.450776862 podStartE2EDuration="22.450776862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:29.450732583 +0000 UTC m=+24.691879089" watchObservedRunningTime="2026-07-15 21:10:29.450776862 +0000 UTC m=+24.691923348" Jul 15 21:10:29.667926 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.667856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:29.770728 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:29.770698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.202370 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.202235 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T21:10:29.438285593Z","UUID":"cd261321-4756-43d7-bc4d-d2bd0739b9ce","Handler":null,"Name":"","Endpoint":""} Jul 15 21:10:30.204197 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.204131 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 21:10:30.204197 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.204165 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 21:10:30.283383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.283348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.287336 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.287313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:30.287477 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:30.287433 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:30.287554 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.287529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:30.287691 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:30.287649 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:30.383930 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.383895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.403904 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.403875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" event={"ID":"39348237-6557-4cce-bea5-eb23ba277a4a","Type":"ContainerStarted","Data":"481386a478ca5b04dd932874e206f8f49469dabaaed63e156725106ce7cf3bcb"} Jul 15 21:10:30.485349 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.485322 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.583880 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.583824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.900593 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:30.900568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:31.287760 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:31.287500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:31.287923 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:31.287872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:31.408925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:31.408889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" event={"ID":"39348237-6557-4cce-bea5-eb23ba277a4a","Type":"ContainerStarted","Data":"844f1a23db9672ede2d6f100dc856f7872913a3bd8edc2ad8cb8330ef4124e94"} Jul 15 21:10:31.412417 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:31.412393 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:10:31.412851 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:31.412821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"f44cda2d3287855cd59ada0ae7af650a85832a9f0b9179411bd863af2150830e"} Jul 15 21:10:31.429390 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:31.429346 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkvv9" podStartSLOduration=3.109863809 podStartE2EDuration="25.429329458s" totalImagesPullingTime="22.319465649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:31.427862767 +0000 UTC m=+26.669009264" watchObservedRunningTime="2026-07-15 21:10:31.429329458 +0000 UTC m=+26.670475946" Jul 15 21:10:32.229775 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.229733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:32.286801 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.286772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:32.286957 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.286807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:32.286957 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:32.286898 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:32.287071 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:32.287027 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:32.322420 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.322386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:32.941882 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.941795 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:32.942641 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:32.942514 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:33.287626 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.287603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:33.287783 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:33.287708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:33.422840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.422820 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:10:33.423274 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.423242 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"0abb338677e2104d3ba34df30f194282daa23ac642c3079f00858bf008a53b09"} Jul 15 21:10:33.423829 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.423808 2574 scope.go:117] "RemoveContainer" containerID="a6f8497dd27ddb227abf9d076c2059ca1bff1214327f9e9d63d2960cc095f40a" Jul 15 21:10:33.424231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.424213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8kgdp" Jul 15 21:10:33.528393 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:33.528198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.287328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.287307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:34.287897 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.287307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:34.287897 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:34.287442 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:34.287897 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:34.287451 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:34.428935 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.428909 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:10:34.429277 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.429258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" event={"ID":"e8622b0a-797d-4b91-917a-ea4f58d59733","Type":"ContainerStarted","Data":"1ba62f9bc22960e470e9e10e0b8ec162a30c2faebd86e7db54f3be6119f51234"} Jul 15 21:10:34.429595 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.429560 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:34.429753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.429642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:34.429753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.429706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:34.431298 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.431270 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="c822428eca409be9bbc51c015b1819a5522587018ee4da4019e0a23aacab4209" exitCode=0 Jul 15 21:10:34.431419 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.431343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"c822428eca409be9bbc51c015b1819a5522587018ee4da4019e0a23aacab4209"} Jul 15 21:10:34.445064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.445044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:34.445219 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.445207 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:10:34.464308 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.464273 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" podStartSLOduration=8.491972117 podStartE2EDuration="28.464260747s" totalImagesPullingTime="19.97228863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:34.462537524 +0000 UTC m=+29.703684024" watchObservedRunningTime="2026-07-15 21:10:34.464260747 +0000 UTC m=+29.705407233" Jul 15 21:10:34.845877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.845845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.908034 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:34.907994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:35.288178 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.288148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:35.288471 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:35.288236 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:35.421452 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.421150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbfsf"] Jul 15 21:10:35.421591 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.421509 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:35.421636 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:35.421602 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:35.425224 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.425180 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6d8qf"] Jul 15 21:10:35.425369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.425315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:35.425450 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:35.425427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:35.425754 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.425734 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nrlr9"] Jul 15 21:10:35.436505 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.436477 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="0b7aa809dc1fe8af3c47268ed53c43f395782e122851cda80995184b7656c6ec" exitCode=0 Jul 15 21:10:35.436657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.436554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"0b7aa809dc1fe8af3c47268ed53c43f395782e122851cda80995184b7656c6ec"} Jul 15 21:10:35.436657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.436605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:35.436797 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:35.436722 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:35.714485 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.714413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:35.907254 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:35.907221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.119777 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.119745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.211772 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.211745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.413920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.413849 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.440339 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.440309 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="06904c9dd8213d28269c6625ff6ea7e63427113c107a7ecc1de96b0d719b6890" exitCode=0 Jul 15 21:10:36.440479 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.440397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"06904c9dd8213d28269c6625ff6ea7e63427113c107a7ecc1de96b0d719b6890"} Jul 15 21:10:36.510801 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:36.510775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.115553 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.115524 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.133630 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.133610 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tb2fz_9469fb72-dc7b-438a-8d69-5c0fc6aff7da/dns-node-resolver/0.log" Jul 15 21:10:37.181633 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.181601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.287626 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.287597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:37.287790 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.287597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:37.287790 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:37.287731 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:37.287790 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.287773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:37.287888 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:37.287851 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:37.287951 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:37.287929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:37.933228 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:37.933200 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s67rl_ce512db5-dc5d-43ff-acd1-5e7f20b33068/node-ca/0.log" Jul 15 21:10:38.055696 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:38.055648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.168651 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:38.168619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.888993 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:38.888956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:38.889182 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.889152 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:38.889182 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.889177 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:38.889467 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.889192 2574 projected.go:196] Error preparing data for projected volume kube-api-access-h6tw5 for pod openshift-network-diagnostics/network-check-target-nrlr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:38.889467 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.889258 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5 podName:48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:11:10.88923914 +0000 UTC m=+66.130385620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6tw5" (UniqueName: "kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5") pod "network-check-target-nrlr9" (UID: "48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:38.989516 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:38.989462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:38.989516 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:38.989517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:38.989957 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.989628 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:38.989957 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.989632 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:38.989957 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.989715 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret podName:64381618-e5ba-4a2d-aeeb-c435bffe9ead nodeName:}" failed. No retries permitted until 2026-07-15 21:11:10.989695316 +0000 UTC m=+66.230841783 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret") pod "global-pull-secret-syncer-vbfsf" (UID: "64381618-e5ba-4a2d-aeeb-c435bffe9ead") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:38.989957 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:38.989748 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs podName:2f49ab87-f4af-4a57-a28c-abb49fd76ca4 nodeName:}" failed. No retries permitted until 2026-07-15 21:11:10.989729565 +0000 UTC m=+66.230876030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs") pod "network-metrics-daemon-6d8qf" (UID: "2f49ab87-f4af-4a57-a28c-abb49fd76ca4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:39.004781 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.004747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:39.287137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.287105 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:39.287310 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.287113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:39.287310 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:39.287242 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:39.287310 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.287119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:39.287474 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:39.287330 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:39.287474 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:39.287390 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:39.738450 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.738335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:39.905924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:39.905882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.010059 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:40.009978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.179706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:40.179657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.289332 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:40.289302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.423853 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:40.423813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:41.208601 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:41.208551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:41.286922 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:41.286886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:41.287071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:41.286885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:41.287071 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:41.287023 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:41.287159 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:41.286885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:41.287159 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:41.287082 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:41.287159 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:41.287147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:41.821142 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:41.821112 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.020177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.020151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.232785 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.232704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.430778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.430734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.455697 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.455656 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="09c1bb2c4928b9a5a919f122a959dbc6db4d237e08ce0432f748df903fe45b8a" exitCode=0 Jul 15 21:10:42.455818 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.455724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"09c1bb2c4928b9a5a919f122a959dbc6db4d237e08ce0432f748df903fe45b8a"} Jul 15 21:10:42.533486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.533461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.629194 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.629150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.744059 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.743963 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.885356 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:42.885329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:43.287955 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.287924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:43.288516 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.287931 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:43.290746 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.287933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:43.290746 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:43.288812 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:43.290746 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:43.289365 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:43.290746 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:43.289780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:43.425585 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.425552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:43.460433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.460407 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87" containerID="3cb12eb9ea4a406a866e8db011cb179cef6d99c73e48a28736a29a74e1fb85b6" exitCode=0 Jul 15 21:10:43.460560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.460447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerDied","Data":"3cb12eb9ea4a406a866e8db011cb179cef6d99c73e48a28736a29a74e1fb85b6"} Jul 15 21:10:43.863404 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:43.863176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:44.001557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.001525 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:44.440049 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.440018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:44.465711 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.465664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" event={"ID":"fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87","Type":"ContainerStarted","Data":"352ab06a401a2ca00547ca833530dde1688736f907c2b1c4f861bf0881478df4"} Jul 15 21:10:44.505834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.505785 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9p2q6" podStartSLOduration=7.191476383 podStartE2EDuration="38.505770609s" totalImagesPullingTime="31.314294226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:44.505575672 +0000 UTC m=+39.746722180" watchObservedRunningTime="2026-07-15 21:10:44.505770609 +0000 UTC m=+39.746917096" Jul 15 21:10:44.775184 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.775101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:44.878169 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:44.878136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:45.234161 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.234126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.288742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:45.288901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.289148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:45.289254 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.289440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:45.291228 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:45.289547 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:45.413068 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.413033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:45.985896 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:45.985846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.105883 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:46.105845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.231192 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:46.231162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.632186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:46.632160 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.848654 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:46.848626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:47.214491 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.214452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:47.286733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.286699 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:47.286733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.286719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:47.286934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.286706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:47.286934 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:47.286827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:47.287035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:47.286927 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:47.287035 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:47.287015 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:47.416577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.416543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:47.608348 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:47.608319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:48.003629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:48.003543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:48.103379 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:48.103329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:48.621297 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:48.621266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:48.928560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:48.928486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:49.030211 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.030162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:49.203256 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.203165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:49.287508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.287477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:49.287508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.287478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:49.287719 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.287503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:49.287719 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:49.287599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:49.287719 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:49.287662 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:49.287831 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:49.287726 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:49.409752 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.409722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:49.614252 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:49.614223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:50.134604 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:50.134573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:50.235413 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:50.235384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:50.298463 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:50.298438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:50.404853 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:50.404785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:51.287243 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:51.287210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:51.287745 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:51.287210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:51.287745 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:51.287311 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:51.287745 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:51.287385 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:51.287745 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:51.287225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:51.287745 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:51.287480 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:53.287226 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:53.287200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:53.287226 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:53.287220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:53.287607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:53.287222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:53.287607 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:53.287289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:53.287607 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:53.287385 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:53.287607 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:53.287437 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:55.290307 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:55.290280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:55.290650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:55.290371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:55.290650 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:55.290378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:55.290650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:55.290395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:55.290650 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:55.290469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:55.290650 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:55.290527 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:57.290258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.290230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:57.290716 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.290238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:57.290716 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:57.290316 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:57.290716 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.290238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:57.290716 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:57.290395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:57.290716 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:57.290459 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:57.325372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.323311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:57.421082 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.421051 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:57.808770 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:57.808744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.032197 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.032172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.099816 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.099744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.459888 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.459811 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.599844 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.599816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.789946 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.789914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.889370 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.889321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.998955 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:58.998928 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:59.247138 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.247042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:59.287094 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.287068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:10:59.287094 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.287088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:10:59.287230 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:59.287164 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrlr9" podUID="48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6" Jul 15 21:10:59.287270 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:59.287236 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d8qf" podUID="2f49ab87-f4af-4a57-a28c-abb49fd76ca4" Jul 15 21:10:59.287270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.287258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:10:59.287342 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:10:59.287330 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbfsf" podUID="64381618-e5ba-4a2d-aeeb-c435bffe9ead" Jul 15 21:10:59.386798 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.386770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:59.552402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.552376 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeReady" Jul 15 21:10:59.552774 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.552502 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 21:10:59.702467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.702436 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lww7q"] Jul 15 21:10:59.704267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.704252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.707907 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.707874 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 21:10:59.707907 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.707891 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 21:10:59.708101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.707926 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 21:10:59.708101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.707874 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 21:10:59.708499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.708481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zghz\"" Jul 15 21:10:59.739853 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.739833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afef0e38-49c4-4f15-ba38-b922166982d4-data-volume\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.739947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.739857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4bv\" (UniqueName: \"kubernetes.io/projected/afef0e38-49c4-4f15-ba38-b922166982d4-kube-api-access-gk4bv\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.739947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.739874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afef0e38-49c4-4f15-ba38-b922166982d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.739947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.739890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afef0e38-49c4-4f15-ba38-b922166982d4-crio-socket\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.739947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.739930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afef0e38-49c4-4f15-ba38-b922166982d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.755404 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.755383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lww7q"] Jul 15 21:10:59.798501 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.798475 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shs46"] Jul 15 21:10:59.800222 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.800206 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sw96t"] Jul 15 21:10:59.800363 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.800345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.802467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.802425 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.803760 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.803745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 21:10:59.804538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.804523 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 21:10:59.815002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.814984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 21:10:59.828118 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.828100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sw96t"] Jul 15 21:10:59.830398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.830375 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 21:10:59.831001 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.830978 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4b2pm\"" Jul 15 21:10:59.831121 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.831098 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 21:10:59.836105 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.836088 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jhvhm\"" Jul 15 21:10:59.836976 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.836959 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shs46"] Jul 15 21:10:59.839872 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.839853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:59.840958 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.840938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afef0e38-49c4-4f15-ba38-b922166982d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841047 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.840967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f277b\" (UniqueName: \"kubernetes.io/projected/1f193cd6-0f42-4248-9524-1561e6e59caa-kube-api-access-f277b\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.841047 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8847d8c3-454a-464e-9eac-52ed4d29a323-metrics-tls\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.841123 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afef0e38-49c4-4f15-ba38-b922166982d4-data-volume\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841123 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8847d8c3-454a-464e-9eac-52ed4d29a323-tmp-dir\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.841123 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4bv\" (UniqueName: \"kubernetes.io/projected/afef0e38-49c4-4f15-ba38-b922166982d4-kube-api-access-gk4bv\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afef0e38-49c4-4f15-ba38-b922166982d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtcm\" (UniqueName: \"kubernetes.io/projected/8847d8c3-454a-464e-9eac-52ed4d29a323-kube-api-access-7dtcm\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.841258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afef0e38-49c4-4f15-ba38-b922166982d4-crio-socket\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8847d8c3-454a-464e-9eac-52ed4d29a323-config-volume\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.841258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f193cd6-0f42-4248-9524-1561e6e59caa-cert\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.841486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afef0e38-49c4-4f15-ba38-b922166982d4-crio-socket\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afef0e38-49c4-4f15-ba38-b922166982d4-data-volume\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.841657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.841635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afef0e38-49c4-4f15-ba38-b922166982d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.845177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.845155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afef0e38-49c4-4f15-ba38-b922166982d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.860578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.860557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4bv\" (UniqueName: \"kubernetes.io/projected/afef0e38-49c4-4f15-ba38-b922166982d4-kube-api-access-gk4bv\") pod \"insights-runtime-extractor-lww7q\" (UID: \"afef0e38-49c4-4f15-ba38-b922166982d4\") " pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:10:59.942006 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.941980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f277b\" (UniqueName: \"kubernetes.io/projected/1f193cd6-0f42-4248-9524-1561e6e59caa-kube-api-access-f277b\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.942168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8847d8c3-454a-464e-9eac-52ed4d29a323-metrics-tls\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.942168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8847d8c3-454a-464e-9eac-52ed4d29a323-tmp-dir\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.942168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtcm\" (UniqueName: \"kubernetes.io/projected/8847d8c3-454a-464e-9eac-52ed4d29a323-kube-api-access-7dtcm\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.942168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8847d8c3-454a-464e-9eac-52ed4d29a323-config-volume\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.942168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f193cd6-0f42-4248-9524-1561e6e59caa-cert\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.942494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8847d8c3-454a-464e-9eac-52ed4d29a323-tmp-dir\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.942791 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.942752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8847d8c3-454a-464e-9eac-52ed4d29a323-config-volume\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.944467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.944444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8847d8c3-454a-464e-9eac-52ed4d29a323-metrics-tls\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:10:59.944571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.944525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f193cd6-0f42-4248-9524-1561e6e59caa-cert\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.964177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.964152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f277b\" (UniqueName: \"kubernetes.io/projected/1f193cd6-0f42-4248-9524-1561e6e59caa-kube-api-access-f277b\") pod \"ingress-canary-shs46\" (UID: \"1f193cd6-0f42-4248-9524-1561e6e59caa\") " pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:10:59.993834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:10:59.993813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtcm\" (UniqueName: \"kubernetes.io/projected/8847d8c3-454a-464e-9eac-52ed4d29a323-kube-api-access-7dtcm\") pod \"dns-default-sw96t\" (UID: \"8847d8c3-454a-464e-9eac-52ed4d29a323\") " pod="openshift-dns/dns-default-sw96t" Jul 15 21:11:00.012793 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.012764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lww7q" Jul 15 21:11:00.109087 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.109060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shs46" Jul 15 21:11:00.116557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.116528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sw96t" Jul 15 21:11:00.200952 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.200925 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lww7q"] Jul 15 21:11:00.204438 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:00.204229 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafef0e38_49c4_4f15_ba38_b922166982d4.slice/crio-ddcd9cd6dc2d2ef92bb005181d5279e9b79934c5c40000918ed11b913f1c3d44 WatchSource:0}: Error finding container ddcd9cd6dc2d2ef92bb005181d5279e9b79934c5c40000918ed11b913f1c3d44: Status 404 returned error can't find the container with id ddcd9cd6dc2d2ef92bb005181d5279e9b79934c5c40000918ed11b913f1c3d44 Jul 15 21:11:00.246038 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.246008 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shs46"] Jul 15 21:11:00.249528 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:00.249505 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f193cd6_0f42_4248_9524_1561e6e59caa.slice/crio-a5e297229bb0ad732e281a6d991c264a0eba1f29c54afc76fef08416120f0196 WatchSource:0}: Error finding container a5e297229bb0ad732e281a6d991c264a0eba1f29c54afc76fef08416120f0196: Status 404 returned error can't find the container with id a5e297229bb0ad732e281a6d991c264a0eba1f29c54afc76fef08416120f0196 Jul 15 21:11:00.264626 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.264606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sw96t"] Jul 15 21:11:00.267272 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:00.267250 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8847d8c3_454a_464e_9eac_52ed4d29a323.slice/crio-bb745d33447dc119872b3d0dcf6f244cedde08ec2808de9eb1412420462d1c47 WatchSource:0}: Error finding container bb745d33447dc119872b3d0dcf6f244cedde08ec2808de9eb1412420462d1c47: Status 404 returned error can't find the container with id bb745d33447dc119872b3d0dcf6f244cedde08ec2808de9eb1412420462d1c47 Jul 15 21:11:00.362154 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.362081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:00.465914 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.465891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:00.500554 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.500527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shs46" event={"ID":"1f193cd6-0f42-4248-9524-1561e6e59caa","Type":"ContainerStarted","Data":"a5e297229bb0ad732e281a6d991c264a0eba1f29c54afc76fef08416120f0196"} Jul 15 21:11:00.501499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.501483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sw96t" event={"ID":"8847d8c3-454a-464e-9eac-52ed4d29a323","Type":"ContainerStarted","Data":"bb745d33447dc119872b3d0dcf6f244cedde08ec2808de9eb1412420462d1c47"} Jul 15 21:11:00.502425 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:00.502410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lww7q" event={"ID":"afef0e38-49c4-4f15-ba38-b922166982d4","Type":"ContainerStarted","Data":"ddcd9cd6dc2d2ef92bb005181d5279e9b79934c5c40000918ed11b913f1c3d44"} Jul 15 21:11:01.287056 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.287021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:11:01.287656 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.287632 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:01.288092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.288070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:11:01.290578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.290558 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xbx8z\"" Jul 15 21:11:01.291905 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.291879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:11:01.292189 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.292167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:11:01.292372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.292331 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:11:01.292372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.292361 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtv82\"" Jul 15 21:11:01.292499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:01.292454 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:11:05.143237 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.143209 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.292103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.292070 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:05.300236 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.300202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.509382 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.509309 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:05.509528 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.509406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.513731 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513700 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 21:11:05.513860 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513779 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 21:11:05.513860 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 21:11:05.513860 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nhpmj\"" Jul 15 21:11:05.514027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513843 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 21:11:05.514027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 21:11:05.514027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.513911 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 21:11:05.514159 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.514142 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 21:11:05.586360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.586360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.586538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.586538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.586538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6nb\" (UniqueName: \"kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.586538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.586484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687286 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687286 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6nb\" (UniqueName: \"kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.687484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.687438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.688068 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.688039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.688068 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.688058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.688233 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.688075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.689722 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.689702 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.689875 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.689857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.697547 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.697526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6nb\" (UniqueName: \"kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb\") pod \"console-57c6d6f88c-d6rfv\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:05.818804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:05.818775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:06.453455 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:06.453437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pjmhv" Jul 15 21:11:06.584550 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:06.584483 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:06.614792 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:06.614763 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.713617 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:06.713542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:07.527715 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.527625 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shs46" event={"ID":"1f193cd6-0f42-4248-9524-1561e6e59caa","Type":"ContainerStarted","Data":"9cf323abbcd4722c2d074ccaa34c55944184868710eb407deac42d95b81d7070"} Jul 15 21:11:07.529607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.529578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sw96t" event={"ID":"8847d8c3-454a-464e-9eac-52ed4d29a323","Type":"ContainerStarted","Data":"4aa44d8804f4d6ffa38d1357de1896508d5b3d50a42666c042188db545fe98a4"} Jul 15 21:11:07.529735 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.529628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sw96t" event={"ID":"8847d8c3-454a-464e-9eac-52ed4d29a323","Type":"ContainerStarted","Data":"08867d2d1f8d26ebad564e1b9385394f374717b7907c716554f2469bf37d63ea"} Jul 15 21:11:07.530000 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.529983 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sw96t" Jul 15 21:11:07.532397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.532375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lww7q" event={"ID":"afef0e38-49c4-4f15-ba38-b922166982d4","Type":"ContainerStarted","Data":"ae4c9f83f37174950f6c2d45f0d9226feb41b75ed406fbf595cb997f820aaad7"} Jul 15 21:11:07.533916 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.533795 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c6d6f88c-d6rfv" event={"ID":"a919c288-e415-4e55-beb7-9c7c167b1b13","Type":"ContainerStarted","Data":"7d996c5efc2914632adebc227e24c278fdfff4cc26490a7740d0f108aab9b68a"} Jul 15 21:11:07.546501 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.546456 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shs46" podStartSLOduration=2.358267002 podStartE2EDuration="8.546443063s" totalImagesPullingTime="6.188176061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:07.545702708 +0000 UTC m=+62.786849205" watchObservedRunningTime="2026-07-15 21:11:07.546443063 +0000 UTC m=+62.787589528" Jul 15 21:11:07.571656 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:07.568998 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sw96t" podStartSLOduration=2.7204503129999997 podStartE2EDuration="8.568980371s" totalImagesPullingTime="5.848530058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:07.567970656 +0000 UTC m=+62.809117144" watchObservedRunningTime="2026-07-15 21:11:07.568980371 +0000 UTC m=+62.810126856" Jul 15 21:11:08.817640 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:08.817597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.128585 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.128477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.232398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.232354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.335383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.335342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.433541 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.433461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.533491 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.533452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.637113 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.637060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.648016 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.647990 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d"] Jul 15 21:11:09.651279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.651262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:09.653950 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.653925 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 21:11:09.654069 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.653990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f54mb\"" Jul 15 21:11:09.660051 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.660034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d"] Jul 15 21:11:09.723600 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.723522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pcq8d\" (UID: \"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:09.824944 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:09.824905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pcq8d\" (UID: \"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:09.825343 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:09.825057 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 15 21:11:09.825343 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:09.825172 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates podName:ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be nodeName:}" failed. No retries permitted until 2026-07-15 21:11:10.325150488 +0000 UTC m=+65.566296977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-pcq8d" (UID: "ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be") : secret "prometheus-operator-admission-webhook-tls" not found Jul 15 21:11:10.160793 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.160754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.263096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.263060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.328817 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.328783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pcq8d\" (UID: \"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:10.331454 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.331424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pcq8d\" (UID: \"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:10.561777 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.561742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:10.935424 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.935338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:10.938565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.938540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:11:10.949507 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.949486 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:11:10.958295 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:10.958269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tw5\" (UniqueName: \"kubernetes.io/projected/48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6-kube-api-access-h6tw5\") pod \"network-check-target-nrlr9\" (UID: \"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6\") " pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:11.036530 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.036499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:11:11.036720 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.036538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:11:11.039757 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.039732 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:11:11.039870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.039766 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:11:11.049850 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.049821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64381618-e5ba-4a2d-aeeb-c435bffe9ead-original-pull-secret\") pod \"global-pull-secret-syncer-vbfsf\" (UID: \"64381618-e5ba-4a2d-aeeb-c435bffe9ead\") " pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:11:11.049947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.049827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f49ab87-f4af-4a57-a28c-abb49fd76ca4-metrics-certs\") pod \"network-metrics-daemon-6d8qf\" (UID: \"2f49ab87-f4af-4a57-a28c-abb49fd76ca4\") " pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:11:11.205096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.205073 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xbx8z\"" Jul 15 21:11:11.212790 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.212741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d8qf" Jul 15 21:11:11.214211 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.214192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtv82\"" Jul 15 21:11:11.219738 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.219720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbfsf" Jul 15 21:11:11.221861 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.221842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:11.223630 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.223609 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d"] Jul 15 21:11:11.226253 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:11.226231 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2244ee_2eb4_48fe_8fa5_20cf5bc0a2be.slice/crio-eb7da04929d906b74d45ed8e1d79de3c6450c6ec5b8a2ccc07d2a9d3431be019 WatchSource:0}: Error finding container eb7da04929d906b74d45ed8e1d79de3c6450c6ec5b8a2ccc07d2a9d3431be019: Status 404 returned error can't find the container with id eb7da04929d906b74d45ed8e1d79de3c6450c6ec5b8a2ccc07d2a9d3431be019 Jul 15 21:11:11.237653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.237620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.413660 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.413504 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6d8qf"] Jul 15 21:11:11.417986 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:11.417955 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f49ab87_f4af_4a57_a28c_abb49fd76ca4.slice/crio-2d579a784f7648b1e9082ba3526b09f2c5c7c677d3d203e8325348c45d0286ab WatchSource:0}: Error finding container 2d579a784f7648b1e9082ba3526b09f2c5c7c677d3d203e8325348c45d0286ab: Status 404 returned error can't find the container with id 2d579a784f7648b1e9082ba3526b09f2c5c7c677d3d203e8325348c45d0286ab Jul 15 21:11:11.440579 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.440555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbfsf"] Jul 15 21:11:11.443851 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:11.443827 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64381618_e5ba_4a2d_aeeb_c435bffe9ead.slice/crio-bde65707c663aaca3196eaf97b9b2a22385e26e7a49c882bb2ec662a2dad4613 WatchSource:0}: Error finding container bde65707c663aaca3196eaf97b9b2a22385e26e7a49c882bb2ec662a2dad4613: Status 404 returned error can't find the container with id bde65707c663aaca3196eaf97b9b2a22385e26e7a49c882bb2ec662a2dad4613 Jul 15 21:11:11.463020 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.462989 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nrlr9"] Jul 15 21:11:11.466564 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:11.466543 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cc9dfe_aac9_43d5_ad4e_08e421f2d8e6.slice/crio-4efdd1f9f37bf4c117ad07d5f8ba4a00d8fb17617ab8d42b63915b0cff85c45b WatchSource:0}: Error finding container 4efdd1f9f37bf4c117ad07d5f8ba4a00d8fb17617ab8d42b63915b0cff85c45b: Status 404 returned error can't find the container with id 4efdd1f9f37bf4c117ad07d5f8ba4a00d8fb17617ab8d42b63915b0cff85c45b Jul 15 21:11:11.545343 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.545316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbfsf" event={"ID":"64381618-e5ba-4a2d-aeeb-c435bffe9ead","Type":"ContainerStarted","Data":"bde65707c663aaca3196eaf97b9b2a22385e26e7a49c882bb2ec662a2dad4613"} Jul 15 21:11:11.546476 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.546453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" event={"ID":"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be","Type":"ContainerStarted","Data":"eb7da04929d906b74d45ed8e1d79de3c6450c6ec5b8a2ccc07d2a9d3431be019"} Jul 15 21:11:11.548125 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.548108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lww7q" event={"ID":"afef0e38-49c4-4f15-ba38-b922166982d4","Type":"ContainerStarted","Data":"9ed53e0f27522d1e194630d2a5611b6b0c7f545d3955fb105ba7be6809469701"} Jul 15 21:11:11.549523 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.549504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c6d6f88c-d6rfv" event={"ID":"a919c288-e415-4e55-beb7-9c7c167b1b13","Type":"ContainerStarted","Data":"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653"} Jul 15 21:11:11.550577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.550558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nrlr9" event={"ID":"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6","Type":"ContainerStarted","Data":"4efdd1f9f37bf4c117ad07d5f8ba4a00d8fb17617ab8d42b63915b0cff85c45b"} Jul 15 21:11:11.551567 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.551549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d8qf" event={"ID":"2f49ab87-f4af-4a57-a28c-abb49fd76ca4","Type":"ContainerStarted","Data":"2d579a784f7648b1e9082ba3526b09f2c5c7c677d3d203e8325348c45d0286ab"} Jul 15 21:11:11.577737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:11.577699 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57c6d6f88c-d6rfv" podStartSLOduration=2.075379363 podStartE2EDuration="6.577667319s" totalImagesPullingTime="4.502287956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:11.57676877 +0000 UTC m=+66.817915281" watchObservedRunningTime="2026-07-15 21:11:11.577667319 +0000 UTC m=+66.818813805" Jul 15 21:11:12.823578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.823340 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:11:12.827607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.827582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.846023 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.846002 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 21:11:12.861091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.860998 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hp9\" (UniqueName: \"kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:12.954377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:12.954260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.054994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hp9\" (UniqueName: \"kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055124 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055124 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055124 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.055260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.056310 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.055926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.056413 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.056382 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.056413 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.056394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.056972 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.056919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.060958 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.060813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.062526 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.062464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.066172 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.066149 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hp9\" (UniqueName: \"kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9\") pod \"console-5cc6b74789-dgvrx\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.142700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.142520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:13.302463 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.302390 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:11:13.306763 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:13.306724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca8edac_4b3a_4dcc_a87a_c6ffd102dbca.slice/crio-0a30ae7677b7a9504f3a3990e9c25e1e954cbca8d00f01c589496dd086d95a5b WatchSource:0}: Error finding container 0a30ae7677b7a9504f3a3990e9c25e1e954cbca8d00f01c589496dd086d95a5b: Status 404 returned error can't find the container with id 0a30ae7677b7a9504f3a3990e9c25e1e954cbca8d00f01c589496dd086d95a5b Jul 15 21:11:13.563155 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.563023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6b74789-dgvrx" event={"ID":"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca","Type":"ContainerStarted","Data":"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d"} Jul 15 21:11:13.563155 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.563084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6b74789-dgvrx" event={"ID":"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca","Type":"ContainerStarted","Data":"0a30ae7677b7a9504f3a3990e9c25e1e954cbca8d00f01c589496dd086d95a5b"} Jul 15 21:11:13.566358 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.565531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" event={"ID":"ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be","Type":"ContainerStarted","Data":"567736454c57b2aef3d0de2c6cfca6901a17a7b6cd10a9f9d44e69770fbdb53c"} Jul 15 21:11:13.566358 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.566093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:13.568272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.568249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d8qf" event={"ID":"2f49ab87-f4af-4a57-a28c-abb49fd76ca4","Type":"ContainerStarted","Data":"b3b6d93bbbeabba5d9177b1855d0308808b0e570f6d00e5061757a62d8c9fd74"} Jul 15 21:11:13.568367 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.568296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d8qf" event={"ID":"2f49ab87-f4af-4a57-a28c-abb49fd76ca4","Type":"ContainerStarted","Data":"3fcf20be18e50cfef43c64e6954e17361ae99c9e9c852b703020042e530413d8"} Jul 15 21:11:13.572177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.572157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" Jul 15 21:11:13.586814 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.586734 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cc6b74789-dgvrx" podStartSLOduration=1.586714986 podStartE2EDuration="1.586714986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:13.585221767 +0000 UTC m=+68.826368255" watchObservedRunningTime="2026-07-15 21:11:13.586714986 +0000 UTC m=+68.827861495" Jul 15 21:11:13.609928 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.609067 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6d8qf" podStartSLOduration=66.070953845 podStartE2EDuration="1m7.609049443s" totalImagesPullingTime="1.538095598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:13.608840518 +0000 UTC m=+68.849987036" watchObservedRunningTime="2026-07-15 21:11:13.609049443 +0000 UTC m=+68.850195927" Jul 15 21:11:13.634806 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.634759 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pcq8d" podStartSLOduration=2.906269932 podStartE2EDuration="4.63474501s" totalImagesPullingTime="1.728475078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:13.63343164 +0000 UTC m=+68.874578168" watchObservedRunningTime="2026-07-15 21:11:13.63474501 +0000 UTC m=+68.875891500" Jul 15 21:11:13.791968 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.791938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:13.897258 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:13.897116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:14.063201 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.063145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:14.163158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.163076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:14.671537 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.671499 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:14.717369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.717335 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-wb8bh"] Jul 15 21:11:14.726513 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.726488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.729897 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.729839 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 21:11:14.731730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.731299 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 21:11:14.731730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.731421 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 21:11:14.731730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.731554 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 21:11:14.731730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.731570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 21:11:14.731730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.731590 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-27gd2\"" Jul 15 21:11:14.733193 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.733156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-wb8bh"] Jul 15 21:11:14.748512 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.748486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:14.769727 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.769662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjhd\" (UniqueName: \"kubernetes.io/projected/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-kube-api-access-xmjhd\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.769727 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.769723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.769881 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.769756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.769881 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.769796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.870486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.870449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjhd\" (UniqueName: \"kubernetes.io/projected/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-kube-api-access-xmjhd\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.870633 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.870497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.870633 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.870534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.870633 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.870567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.871386 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.871360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.873221 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.873198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.873371 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.873345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.882184 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.882150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjhd\" (UniqueName: \"kubernetes.io/projected/b3d783de-ce9b-4904-ae2b-23d5746e2ed9-kube-api-access-xmjhd\") pod \"prometheus-operator-64785b6675-wb8bh\" (UID: \"b3d783de-ce9b-4904-ae2b-23d5746e2ed9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:14.895268 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:14.895241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:15.040367 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:15.040321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" Jul 15 21:11:15.819610 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:15.819579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:15.819804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:15.819644 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:15.825737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:15.825712 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:16.412792 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:16.412751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:16.519438 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:16.519407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:16.583744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:16.583713 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:17.100294 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:17.100223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:17.198076 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:17.198039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:17.538770 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:17.538735 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sw96t" Jul 15 21:11:18.109505 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.109430 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-wb8bh"] Jul 15 21:11:18.112502 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:18.112474 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d783de_ce9b_4904_ae2b_23d5746e2ed9.slice/crio-45afe71eedd92f5687b128f62df9cef023f0b957f26b814de2598a12a764d85e WatchSource:0}: Error finding container 45afe71eedd92f5687b128f62df9cef023f0b957f26b814de2598a12a764d85e: Status 404 returned error can't find the container with id 45afe71eedd92f5687b128f62df9cef023f0b957f26b814de2598a12a764d85e Jul 15 21:11:18.244531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.244456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:18.310941 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.310908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:18.588896 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.588864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nrlr9" event={"ID":"48cc9dfe-aac9-43d5-ad4e-08e421f2d8e6","Type":"ContainerStarted","Data":"7bc8875c1038a58235c9d7ece2962ac2f40853173095246d2d961876a04ad113"} Jul 15 21:11:18.589326 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.588975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:18.590647 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.590614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbfsf" event={"ID":"64381618-e5ba-4a2d-aeeb-c435bffe9ead","Type":"ContainerStarted","Data":"ccfad1d9d462d540eaf338089b72626eed1d53babf6021db430fdffc1d73a644"} Jul 15 21:11:18.591854 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.591825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" event={"ID":"b3d783de-ce9b-4904-ae2b-23d5746e2ed9","Type":"ContainerStarted","Data":"45afe71eedd92f5687b128f62df9cef023f0b957f26b814de2598a12a764d85e"} Jul 15 21:11:18.594021 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.593998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lww7q" event={"ID":"afef0e38-49c4-4f15-ba38-b922166982d4","Type":"ContainerStarted","Data":"39dc031c41eaf20ba5d53d658758554e152780a51d2a3aa219a8124ce875b9ce"} Jul 15 21:11:18.610329 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.610108 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nrlr9" podStartSLOduration=66.098343403 podStartE2EDuration="1m12.610095866s" totalImagesPullingTime="6.511752463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:18.609314589 +0000 UTC m=+73.850461076" watchObservedRunningTime="2026-07-15 21:11:18.610095866 +0000 UTC m=+73.851242398" Jul 15 21:11:18.662919 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.662865 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lww7q" podStartSLOduration=2.134989582 podStartE2EDuration="19.662850801s" totalImagesPullingTime="17.527861219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:18.66258846 +0000 UTC m=+73.903734969" watchObservedRunningTime="2026-07-15 21:11:18.662850801 +0000 UTC m=+73.903997290" Jul 15 21:11:18.663376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:18.663345 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vbfsf" podStartSLOduration=66.134661405 podStartE2EDuration="1m12.663337999s" totalImagesPullingTime="6.528676594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:18.633089011 +0000 UTC m=+73.874235497" watchObservedRunningTime="2026-07-15 21:11:18.663337999 +0000 UTC m=+73.904484485" Jul 15 21:11:19.251043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.250989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:19.352222 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.352198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:19.599302 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.599273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" event={"ID":"b3d783de-ce9b-4904-ae2b-23d5746e2ed9","Type":"ContainerStarted","Data":"d500b8e182e9fbb6fff0f83cf3d8d0b164c9da1086f4a7821ef73196587be15d"} Jul 15 21:11:19.599695 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.599323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" event={"ID":"b3d783de-ce9b-4904-ae2b-23d5746e2ed9","Type":"ContainerStarted","Data":"76e29fbc798f6d50b3af538ee2a9d5b99bab2a7e1c2f452eaba42394f617b9a7"} Jul 15 21:11:19.768388 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.768352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:19.869586 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:19.869502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:20.185778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:20.185703 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:20.278698 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:20.278650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:21.825413 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:21.825382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.044584 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.044551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.151132 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.151041 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-wb8bh" podStartSLOduration=6.944621908 podStartE2EDuration="8.151010518s" totalImagesPullingTime="1.20638861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:19.661750378 +0000 UTC m=+74.902896876" watchObservedRunningTime="2026-07-15 21:11:22.151010518 +0000 UTC m=+77.392157005" Jul 15 21:11:22.151416 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.151401 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2rsxl"] Jul 15 21:11:22.154739 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.154714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.157319 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.157298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.160010 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.159992 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 21:11:22.160832 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.160780 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 21:11:22.161708 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.161689 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6gx47\"" Jul 15 21:11:22.161778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.161705 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 21:11:22.227070 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227114 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-metrics-client-ca\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-sys\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-root\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggrg\" (UniqueName: \"kubernetes.io/projected/4354908d-9298-47c3-8ed2-d15b99b2f47f-kube-api-access-nggrg\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-wtmp\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-textfile\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.227335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.227325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.264482 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.264457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.327806 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-metrics-client-ca\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-sys\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-root\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggrg\" (UniqueName: \"kubernetes.io/projected/4354908d-9298-47c3-8ed2-d15b99b2f47f-kube-api-access-nggrg\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-sys\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-root\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.327924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-wtmp\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:22.327950 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-wtmp\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:22.328007 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls podName:4354908d-9298-47c3-8ed2-d15b99b2f47f nodeName:}" failed. No retries permitted until 2026-07-15 21:11:22.827989404 +0000 UTC m=+78.069135893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls") pod "node-exporter-2rsxl" (UID: "4354908d-9298-47c3-8ed2-d15b99b2f47f") : secret "node-exporter-tls" not found Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.327952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-textfile\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.328036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.328056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.328244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-textfile\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328575 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.328495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-metrics-client-ca\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.328575 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.328522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.330234 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.330216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.357719 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.357694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggrg\" (UniqueName: \"kubernetes.io/projected/4354908d-9298-47c3-8ed2-d15b99b2f47f-kube-api-access-nggrg\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.360216 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.360193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.464326 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.464265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.569467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.569437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.659386 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.659354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.762242 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.762161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:22.832626 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.832595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.834805 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.834786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4354908d-9298-47c3-8ed2-d15b99b2f47f-node-exporter-tls\") pod \"node-exporter-2rsxl\" (UID: \"4354908d-9298-47c3-8ed2-d15b99b2f47f\") " pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:22.909529 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:22.909501 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:23.064268 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.064235 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rsxl" Jul 15 21:11:23.076753 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:23.076728 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354908d_9298_47c3_8ed2_d15b99b2f47f.slice/crio-19063cd03d0b874c54bc7f63a2cf7763390807d8f96305777568378f8aa2131f WatchSource:0}: Error finding container 19063cd03d0b874c54bc7f63a2cf7763390807d8f96305777568378f8aa2131f: Status 404 returned error can't find the container with id 19063cd03d0b874c54bc7f63a2cf7763390807d8f96305777568378f8aa2131f Jul 15 21:11:23.119930 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.119903 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:23.143394 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.143372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:23.143505 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.143426 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:23.148213 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.148195 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:23.205864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.205837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:23.209157 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.209132 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:23.213535 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.213516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.217549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.216321 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 21:11:23.217549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.216608 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 21:11:23.217549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.216913 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 21:11:23.217549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217105 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 21:11:23.217549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217538 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 21:11:23.217893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217592 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 21:11:23.217893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 21:11:23.217893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217620 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2nrzq\"" Jul 15 21:11:23.217893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 21:11:23.217893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.217663 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 21:11:23.234070 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.234050 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:23.337754 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337754 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-config-volume\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-config-out\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.337920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.337949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.338010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.338041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7z2t\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-kube-api-access-l7z2t\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.338063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.338080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.338120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.338100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-web-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439093 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-config-volume\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-config-out\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439314 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439619 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:23.439435 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 21:11:23.439619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439619 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:23.439594 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls podName:08d88ef0-1ff1-4dfc-a820-995124843411 nodeName:}" failed. No retries permitted until 2026-07-15 21:11:23.93948643 +0000 UTC m=+79.180632910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "08d88ef0-1ff1-4dfc-a820-995124843411") : secret "alertmanager-main-tls" not found Jul 15 21:11:23.439838 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7z2t\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-kube-api-access-l7z2t\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439838 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439838 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.439838 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-web-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.440030 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.439996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.440731 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.440660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.440857 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.440812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d88ef0-1ff1-4dfc-a820-995124843411-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442141 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442295 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442380 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442380 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-web-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-config-volume\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.442841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.442830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08d88ef0-1ff1-4dfc-a820-995124843411-config-out\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.443128 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.443108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.444213 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.444192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.449216 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.449193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7z2t\" (UniqueName: \"kubernetes.io/projected/08d88ef0-1ff1-4dfc-a820-995124843411-kube-api-access-l7z2t\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.615475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.615387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rsxl" event={"ID":"4354908d-9298-47c3-8ed2-d15b99b2f47f","Type":"ContainerStarted","Data":"19063cd03d0b874c54bc7f63a2cf7763390807d8f96305777568378f8aa2131f"} Jul 15 21:11:23.620820 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.620792 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:11:23.670078 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.670052 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:23.793318 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.793286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:23.944618 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.944549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:23.946992 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:23.946966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08d88ef0-1ff1-4dfc-a820-995124843411-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08d88ef0-1ff1-4dfc-a820-995124843411\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:24.123734 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.123700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:24.252078 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.252046 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:24.256311 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.256284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:24.261822 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:24.259496 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d88ef0_1ff1_4dfc_a820_995124843411.slice/crio-0848e4797c9f14ad8dc9d6b4db45033971a88c5b99e3676b23a9ec0706e1294d WatchSource:0}: Error finding container 0848e4797c9f14ad8dc9d6b4db45033971a88c5b99e3676b23a9ec0706e1294d: Status 404 returned error can't find the container with id 0848e4797c9f14ad8dc9d6b4db45033971a88c5b99e3676b23a9ec0706e1294d Jul 15 21:11:24.332091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.332044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:24.619307 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.619267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"0848e4797c9f14ad8dc9d6b4db45033971a88c5b99e3676b23a9ec0706e1294d"} Jul 15 21:11:24.620765 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.620744 2574 generic.go:356] "Generic (PLEG): container finished" podID="4354908d-9298-47c3-8ed2-d15b99b2f47f" containerID="4eacc1ed44f74170b8a344e022615e9e7fee5f0a63ba6cdad4700ee95c3d7f50" exitCode=0 Jul 15 21:11:24.620854 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.620815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rsxl" event={"ID":"4354908d-9298-47c3-8ed2-d15b99b2f47f","Type":"ContainerDied","Data":"4eacc1ed44f74170b8a344e022615e9e7fee5f0a63ba6cdad4700ee95c3d7f50"} Jul 15 21:11:24.859222 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.859185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:24.962199 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:24.962092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:25.072411 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.072378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:25.174313 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.174281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:25.376103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.376070 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:25.477891 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.477816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:25.625898 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.625864 2574 generic.go:356] "Generic (PLEG): container finished" podID="08d88ef0-1ff1-4dfc-a820-995124843411" containerID="235cb5b63b0c9e8db1c148b11a4dc0562d66efcce98b5a0b5a07ccb0b694e1a4" exitCode=0 Jul 15 21:11:25.626068 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.625956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerDied","Data":"235cb5b63b0c9e8db1c148b11a4dc0562d66efcce98b5a0b5a07ccb0b694e1a4"} Jul 15 21:11:25.628139 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.628120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rsxl" event={"ID":"4354908d-9298-47c3-8ed2-d15b99b2f47f","Type":"ContainerStarted","Data":"95523a770009ad7e03dd3f30ea78e6cd8ff70f8c1da2750fe4c3f8bf955c3158"} Jul 15 21:11:25.628236 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.628159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rsxl" event={"ID":"4354908d-9298-47c3-8ed2-d15b99b2f47f","Type":"ContainerStarted","Data":"10907a0ca996beb74f9bc90b421ede6d2d0a7f745a8c5a35581e82b6b6566cd4"} Jul 15 21:11:25.673665 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:25.673609 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2rsxl" podStartSLOduration=2.688197487 podStartE2EDuration="3.67359519s" totalImagesPullingTime="922.227076ms" totalInitContainerRuntime="63.170627ms" isStatefulPod=true podCreationTimestamp="2026-07-15 21:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:25.67269262 +0000 UTC m=+80.913839126" watchObservedRunningTime="2026-07-15 21:11:25.67359519 +0000 UTC m=+80.914741676" Jul 15 21:11:26.652909 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.652878 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57746d6954-p74xg"] Jul 15 21:11:26.655908 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.655885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.659888 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.659862 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 21:11:26.659888 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.659862 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-129ph59ah02a9\"" Jul 15 21:11:26.660027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.659972 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 21:11:26.660186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.660168 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 21:11:26.660235 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.660168 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 21:11:26.660487 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.660469 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bt2bd\"" Jul 15 21:11:26.678288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.678262 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57746d6954-p74xg"] Jul 15 21:11:26.766381 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-audit-log\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766552 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766552 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-tls\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-client-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88frq\" (UniqueName: \"kubernetes.io/projected/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-kube-api-access-88frq\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-metrics-server-audit-profiles\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.766710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.766704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-client-certs\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.859861 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.859783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:26.867914 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.867886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.867922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-tls\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.867940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-client-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.867967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88frq\" (UniqueName: \"kubernetes.io/projected/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-kube-api-access-88frq\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.867994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-metrics-server-audit-profiles\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.868118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-client-certs\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.868246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-audit-log\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868741 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.868718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-audit-log\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.868842 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.868727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.869315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.869285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-metrics-server-audit-profiles\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.870707 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.870665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-tls\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.870929 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.870900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-secret-metrics-server-client-certs\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.871036 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.871008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-client-ca-bundle\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.881734 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.881711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88frq\" (UniqueName: \"kubernetes.io/projected/0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c-kube-api-access-88frq\") pod \"metrics-server-57746d6954-p74xg\" (UID: \"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c\") " pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.965630 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.965471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:26.980475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:26.980428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:27.131313 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.131290 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57746d6954-p74xg"] Jul 15 21:11:27.134023 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:27.133995 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0109c5b5_ccb1_45ed_9f8d_f0b33e00f41c.slice/crio-19bfc1bfd590f17b6907c1ea64aa03c78c57fa27b41c07a82bcb16148e35fd56 WatchSource:0}: Error finding container 19bfc1bfd590f17b6907c1ea64aa03c78c57fa27b41c07a82bcb16148e35fd56: Status 404 returned error can't find the container with id 19bfc1bfd590f17b6907c1ea64aa03c78c57fa27b41c07a82bcb16148e35fd56 Jul 15 21:11:27.296171 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.296008 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:27.641338 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.641300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"1b04cc1d15b97a1ef4ecd5ea7b779ad1bf3e815fea25f5fe14b07bf5e94bb484"} Jul 15 21:11:27.641503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.641379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"445f3ac4586ed62dd57d7623a621f4873d4c9b440fa130aa7cf3611ac06535f6"} Jul 15 21:11:27.641503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.641416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"629833938c8763b63f96311533a7096baf5aae5afa1114dd622e1a3fd1f0aff9"} Jul 15 21:11:27.641503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.641451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"3d55b5e717b3ae4e679c7422363abadb2bd152826a2bbc89397bf76440960b7c"} Jul 15 21:11:27.641503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.641488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"7b49cede3c8859f518d40bd66ff088e916d5eb75fd160a0c181394f3075ddf0d"} Jul 15 21:11:27.643347 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.643264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" event={"ID":"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c","Type":"ContainerStarted","Data":"19bfc1bfd590f17b6907c1ea64aa03c78c57fa27b41c07a82bcb16148e35fd56"} Jul 15 21:11:27.857717 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:27.857667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:28.212638 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.212604 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:28.329696 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.329647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:28.360105 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.360068 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:28.366187 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.365205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.369361 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.368876 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 21:11:28.369361 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.368909 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 21:11:28.369361 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.369043 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 21:11:28.372909 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.372884 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 21:11:28.373148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.373124 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmbj4n359bqls\"" Jul 15 21:11:28.375198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.374836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 21:11:28.375198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.374898 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nnt4q\"" Jul 15 21:11:28.375198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.374909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 21:11:28.375198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.375052 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 21:11:28.375198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.375086 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 21:11:28.375485 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.375463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 21:11:28.376795 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.376389 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 21:11:28.376890 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.376821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 21:11:28.376890 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.376889 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 21:11:28.377919 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.377810 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 21:11:28.380654 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.380629 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 21:11:28.386099 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.386076 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:28.418151 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.418116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8ch\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-kube-api-access-xg8ch\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.481974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.482007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.482049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.482076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.482864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.482110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583384 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8ch\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-kube-api-access-xg8ch\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.583953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.583937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.584532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.584306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.585002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.584707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589193 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.587463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589193 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.588161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589193 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.588861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589411 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.589292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589736 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.589651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.589736 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.589698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d009018d-540d-4dbd-9eca-8df5283d234c-config-out\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.590076 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.589609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.590233 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.590212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.590722 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.590577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.590722 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.590624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.591208 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.591140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d009018d-540d-4dbd-9eca-8df5283d234c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.591711 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.591650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-web-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.594376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.593539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.594376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.593737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.594376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.594311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.594967 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.594939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-config\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.595931 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.595907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d009018d-540d-4dbd-9eca-8df5283d234c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.602237 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.602196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8ch\" (UniqueName: \"kubernetes.io/projected/d009018d-540d-4dbd-9eca-8df5283d234c-kube-api-access-xg8ch\") pod \"prometheus-k8s-0\" (UID: \"d009018d-540d-4dbd-9eca-8df5283d234c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.681353 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.681328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:28.831870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:28.831788 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:28.834404 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:11:28.834378 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd009018d_540d_4dbd_9eca_8df5283d234c.slice/crio-aa12d40c0c887293909e4c54b60623b3cdbe044c9451f6288049642a4fdd8a70 WatchSource:0}: Error finding container aa12d40c0c887293909e4c54b60623b3cdbe044c9451f6288049642a4fdd8a70: Status 404 returned error can't find the container with id aa12d40c0c887293909e4c54b60623b3cdbe044c9451f6288049642a4fdd8a70 Jul 15 21:11:29.653074 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.653043 2574 generic.go:356] "Generic (PLEG): container finished" podID="d009018d-540d-4dbd-9eca-8df5283d234c" containerID="f8616a70ce6dd5b2b3aac3c978d7d9b21c3f8e5f552c406484c51bc6563f136f" exitCode=0 Jul 15 21:11:29.653514 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.653139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerDied","Data":"f8616a70ce6dd5b2b3aac3c978d7d9b21c3f8e5f552c406484c51bc6563f136f"} Jul 15 21:11:29.653514 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.653220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"aa12d40c0c887293909e4c54b60623b3cdbe044c9451f6288049642a4fdd8a70"} Jul 15 21:11:29.657003 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.656978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08d88ef0-1ff1-4dfc-a820-995124843411","Type":"ContainerStarted","Data":"55ff6020c164aa32bc9858a26eea9cd75bc0eb7cc3242ae8f10da7a357e5de30"} Jul 15 21:11:29.658636 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.658611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" event={"ID":"0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c","Type":"ContainerStarted","Data":"b6cb918fdea2adc79d07abdc408833a74de5c217bd05e5d428046affa73bd686"} Jul 15 21:11:29.716481 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.716421 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.108491809 podStartE2EDuration="6.716401733s" totalImagesPullingTime="3.592048956s" totalInitContainerRuntime="15.860968ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:29.714019992 +0000 UTC m=+84.955166479" watchObservedRunningTime="2026-07-15 21:11:29.716401733 +0000 UTC m=+84.957548220" Jul 15 21:11:29.737110 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:29.737053 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" podStartSLOduration=2.202072069 podStartE2EDuration="3.737033565s" totalImagesPullingTime="1.534961496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:29.736695209 +0000 UTC m=+84.977841699" watchObservedRunningTime="2026-07-15 21:11:29.737033565 +0000 UTC m=+84.978180053" Jul 15 21:11:32.139773 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:32.139720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:32.242062 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:32.242004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:32.675752 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:32.675716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"4bd65054ab0f8ed5f44cc44d7cdbc9b07c9abbdddab1189a90e3af129368b6d2"} Jul 15 21:11:32.675913 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:32.675785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"0187d50809ab5f4d1fdd4b11b2131a73772c55ee75d530bf4d626014bc5fc30b"} Jul 15 21:11:34.386649 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:34.386607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:34.452863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:34.452818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:35.127935 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.127905 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:35.227782 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.227754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:35.691667 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.691634 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"46d707dbabec8e1507e98bdd1975e4c3ccafe9b7167c071947c73611161288fd"} Jul 15 21:11:35.692153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.691723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"2e4298011207fafa4a8a55c69a7807166761fd087ac42f5a604b87f79501ab24"} Jul 15 21:11:35.692153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.691765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"81c4b3feae13c1ae1e765628b299188f3c519fe21b5bea20f1dcb394452069e9"} Jul 15 21:11:35.692153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.691807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d009018d-540d-4dbd-9eca-8df5283d234c","Type":"ContainerStarted","Data":"23d281f2b1201c8fdd37db6f42e7fe0b40a10ad72845734ecb46f30e20158487"} Jul 15 21:11:35.726273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:35.726221 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.551681223 podStartE2EDuration="7.726205469s" totalImagesPullingTime="5.154394504s" totalInitContainerRuntime="20.129742ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:35.723819427 +0000 UTC m=+90.964965935" watchObservedRunningTime="2026-07-15 21:11:35.726205469 +0000 UTC m=+90.967351958" Jul 15 21:11:38.682046 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:38.682009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:43.495326 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:43.495289 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:11:43.637759 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:43.637723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:43.699424 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:43.699391 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:43.761644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:43.761570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:46.966343 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:46.966308 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:46.966812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:46.966364 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:11:48.693865 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:48.693800 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57c6d6f88c-d6rfv" podUID="a919c288-e415-4e55-beb7-9c7c167b1b13" containerName="console" containerID="cri-o://53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653" gracePeriod=15 Jul 15 21:11:49.463858 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.463831 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:49.573078 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573043 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm6nb\" (UniqueName: \"kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573140 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573178 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573259 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573229 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config\") pod \"a919c288-e415-4e55-beb7-9c7c167b1b13\" (UID: \"a919c288-e415-4e55-beb7-9c7c167b1b13\") " Jul 15 21:11:49.573571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:49.573634 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573564 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca" (OuterVolumeSpecName: "service-ca") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:49.573701 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.573638 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config" (OuterVolumeSpecName: "console-config") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:49.575568 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.575546 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:11:49.575649 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.575579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb" (OuterVolumeSpecName: "kube-api-access-bm6nb") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "kube-api-access-bm6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:11:49.575649 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.575586 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a919c288-e415-4e55-beb7-9c7c167b1b13" (UID: "a919c288-e415-4e55-beb7-9c7c167b1b13"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:11:49.601540 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.601518 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nrlr9" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674150 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-oauth-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674171 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6nb\" (UniqueName: \"kubernetes.io/projected/a919c288-e415-4e55-beb7-9c7c167b1b13-kube-api-access-bm6nb\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674181 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674189 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674199 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a919c288-e415-4e55-beb7-9c7c167b1b13-console-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.674214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.674207 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a919c288-e415-4e55-beb7-9c7c167b1b13-console-oauth-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:11:49.746997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.746963 2574 generic.go:356] "Generic (PLEG): container finished" podID="a919c288-e415-4e55-beb7-9c7c167b1b13" containerID="53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653" exitCode=0 Jul 15 21:11:49.747415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.747010 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c6d6f88c-d6rfv" Jul 15 21:11:49.747415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.747047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c6d6f88c-d6rfv" event={"ID":"a919c288-e415-4e55-beb7-9c7c167b1b13","Type":"ContainerDied","Data":"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653"} Jul 15 21:11:49.747415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.747104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c6d6f88c-d6rfv" event={"ID":"a919c288-e415-4e55-beb7-9c7c167b1b13","Type":"ContainerDied","Data":"7d996c5efc2914632adebc227e24c278fdfff4cc26490a7740d0f108aab9b68a"} Jul 15 21:11:49.747415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.747135 2574 scope.go:117] "RemoveContainer" containerID="53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653" Jul 15 21:11:49.758415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.758400 2574 scope.go:117] "RemoveContainer" containerID="53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653" Jul 15 21:11:49.758665 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:11:49.758646 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653\": container with ID starting with 53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653 not found: ID does not exist" containerID="53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653" Jul 15 21:11:49.758726 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.758686 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653"} err="failed to get container status \"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653\": rpc error: code = NotFound desc = could not find container \"53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653\": container with ID starting with 53e43fcf615c446c99437df478612acdee5bb318e068c390a20d0e00878ef653 not found: ID does not exist" Jul 15 21:11:49.769812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.769788 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:49.773375 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:49.773354 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57c6d6f88c-d6rfv"] Jul 15 21:11:50.135122 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:50.135085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:50.213307 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:50.213277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:50.327623 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:50.327586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:50.413081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:50.412983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:51.291565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:11:51.291532 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a919c288-e415-4e55-beb7-9c7c167b1b13" path="/var/lib/kubelet/pods/a919c288-e415-4e55-beb7-9c7c167b1b13/volumes" Jul 15 21:12:06.971279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:06.971251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:12:06.974887 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:06.974866 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57746d6954-p74xg" Jul 15 21:12:08.515370 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.515313 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cc6b74789-dgvrx" podUID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" containerName="console" containerID="cri-o://5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d" gracePeriod=15 Jul 15 21:12:08.741762 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.741740 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:12:08.822039 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.822015 2574 generic.go:356] "Generic (PLEG): container finished" podID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" containerID="5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d" exitCode=0 Jul 15 21:12:08.822167 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.822060 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6b74789-dgvrx" Jul 15 21:12:08.822167 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.822072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6b74789-dgvrx" event={"ID":"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca","Type":"ContainerDied","Data":"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d"} Jul 15 21:12:08.822167 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.822117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6b74789-dgvrx" event={"ID":"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca","Type":"ContainerDied","Data":"0a30ae7677b7a9504f3a3990e9c25e1e954cbca8d00f01c589496dd086d95a5b"} Jul 15 21:12:08.822167 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.822139 2574 scope.go:117] "RemoveContainer" containerID="5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d" Jul 15 21:12:08.827280 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827302 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827334 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827524 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827404 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42hp9\" (UniqueName: \"kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827524 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827524 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config\") pod \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\" (UID: \"4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca\") " Jul 15 21:12:08.827710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827669 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:08.827831 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827733 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:08.827894 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827873 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca" (OuterVolumeSpecName: "service-ca") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:08.827954 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.827926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config" (OuterVolumeSpecName: "console-config") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:08.829805 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.829772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9" (OuterVolumeSpecName: "kube-api-access-42hp9") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "kube-api-access-42hp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:12:08.829805 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.829772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:08.829968 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.829886 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" (UID: "4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:08.830522 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.830503 2574 scope.go:117] "RemoveContainer" containerID="5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d" Jul 15 21:12:08.830768 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:12:08.830749 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d\": container with ID starting with 5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d not found: ID does not exist" containerID="5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d" Jul 15 21:12:08.830833 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.830776 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d"} err="failed to get container status \"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d\": rpc error: code = NotFound desc = could not find container \"5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d\": container with ID starting with 5a764c02144df19d488ad043af7ff032caed21d0f5d1bc7d9f10cd1472844e1d not found: ID does not exist" Jul 15 21:12:08.928763 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928742 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928763 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928761 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-oauth-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928771 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-trusted-ca-bundle\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928780 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928791 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hp9\" (UniqueName: \"kubernetes.io/projected/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-kube-api-access-42hp9\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928804 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:08.928877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:08.928812 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca-console-oauth-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:12:09.149975 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:09.149941 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:12:09.157284 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:09.157264 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cc6b74789-dgvrx"] Jul 15 21:12:09.291557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:09.291521 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" path="/var/lib/kubelet/pods/4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca/volumes" Jul 15 21:12:28.681530 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:28.681494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:28.701153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:28.701128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:28.914976 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:28.914951 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:38.252689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:38.252639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:38.325844 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:38.325809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:50.050943 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:50.050916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:50.109602 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:50.109570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:55.850992 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:55.850956 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:57.580228 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:57.580153 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:57.680696 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:57.680653 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:57.780975 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:57.780944 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:58.328255 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:58.328222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:58.428860 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:12:58.428830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:05.152200 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:05.152158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:05.251553 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:05.251524 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:08.946656 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:08.946625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:09.047374 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:09.047347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:09.105369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:09.105336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:10.021774 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:10.021740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:21.575301 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:21.575268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:21.676024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:21.675992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:34.186162 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:34.186130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:34.287323 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:34.287288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:55.901807 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:55.901769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:56.702492 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:56.702458 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:59.721820 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:59.721790 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:59.824230 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:13:59.824198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:00.952374 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:00.952342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:01.048209 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.048174 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:01.380774 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.380743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:01.478183 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.478157 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:01.922489 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922460 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j5bzp"] Jul 15 21:14:01.922742 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922730 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" containerName="console" Jul 15 21:14:01.922788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922743 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" containerName="console" Jul 15 21:14:01.922788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922754 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a919c288-e415-4e55-beb7-9c7c167b1b13" containerName="console" Jul 15 21:14:01.922788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922759 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a919c288-e415-4e55-beb7-9c7c167b1b13" containerName="console" Jul 15 21:14:01.922910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922806 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca8edac-4b3a-4dcc-a87a-c6ffd102dbca" containerName="console" Jul 15 21:14:01.922910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.922813 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a919c288-e415-4e55-beb7-9c7c167b1b13" containerName="console" Jul 15 21:14:01.925477 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.925461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:01.928351 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.928328 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 21:14:01.928484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.928382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xh6vt\"" Jul 15 21:14:01.929804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.929786 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 21:14:01.934336 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.934317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j5bzp"] Jul 15 21:14:01.971771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.971745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779b4\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-kube-api-access-779b4\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:01.972123 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:01.971773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.072806 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.072778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779b4\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-kube-api-access-779b4\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.072806 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.072807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.081081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.081058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.081196 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.081181 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779b4\" (UniqueName: \"kubernetes.io/projected/cd18dfb6-465d-4c96-b3f2-431d6af25fad-kube-api-access-779b4\") pod \"cert-manager-webhook-6486ffd997-j5bzp\" (UID: \"cd18dfb6-465d-4c96-b3f2-431d6af25fad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.254561 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.254482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:02.380909 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.380741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j5bzp"] Jul 15 21:14:02.383583 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:14:02.383557 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd18dfb6_465d_4c96_b3f2_431d6af25fad.slice/crio-35322902aa62b0db11392a72ad74bfe5ab833d6734fc1da7df6ba3202cc221ba WatchSource:0}: Error finding container 35322902aa62b0db11392a72ad74bfe5ab833d6734fc1da7df6ba3202cc221ba: Status 404 returned error can't find the container with id 35322902aa62b0db11392a72ad74bfe5ab833d6734fc1da7df6ba3202cc221ba Jul 15 21:14:02.961887 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:02.961859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:03.226013 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:03.225939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" event={"ID":"cd18dfb6-465d-4c96-b3f2-431d6af25fad","Type":"ContainerStarted","Data":"35322902aa62b0db11392a72ad74bfe5ab833d6734fc1da7df6ba3202cc221ba"} Jul 15 21:14:03.349710 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:03.349664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:06.239339 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:06.239251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" event={"ID":"cd18dfb6-465d-4c96-b3f2-431d6af25fad","Type":"ContainerStarted","Data":"3c0402647da4e7f983ccaddf9ab0a496dbb76256093365f365327f68e21e57c6"} Jul 15 21:14:06.239719 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:06.239372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:06.507630 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:06.507519 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:06.608232 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:06.608200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:07.312003 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:07.311973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:07.414689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:07.414651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:11.550936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.550905 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:11.712387 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.712335 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" podStartSLOduration=7.124240058 podStartE2EDuration="10.712319037s" totalImagesPullingTime="3.588078979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:14:06.258125257 +0000 UTC m=+241.499271743" watchObservedRunningTime="2026-07-15 21:14:11.712319037 +0000 UTC m=+246.953465523" Jul 15 21:14:11.712745 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.712732 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sjtm8"] Jul 15 21:14:11.715852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.715838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.718449 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.718423 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pxj9l\"" Jul 15 21:14:11.723828 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.723804 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sjtm8"] Jul 15 21:14:11.850546 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.850455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.850546 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.850495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt8l\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-kube-api-access-tlt8l\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.951703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.951653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.951805 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.951708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt8l\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-kube-api-access-tlt8l\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.960163 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.960136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:11.960354 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:11.960334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt8l\" (UniqueName: \"kubernetes.io/projected/1f975d04-037c-40d6-b6f6-8283201121b4-kube-api-access-tlt8l\") pod \"cert-manager-55b4d954d5-sjtm8\" (UID: \"1f975d04-037c-40d6-b6f6-8283201121b4\") " pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:12.024945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.024921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sjtm8" Jul 15 21:14:12.149534 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.149376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sjtm8"] Jul 15 21:14:12.152183 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:14:12.152156 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f975d04_037c_40d6_b6f6_8283201121b4.slice/crio-f052b844e2befaefde446b2832a835a7caa309f479e35e015d245ce6812db0ff WatchSource:0}: Error finding container f052b844e2befaefde446b2832a835a7caa309f479e35e015d245ce6812db0ff: Status 404 returned error can't find the container with id f052b844e2befaefde446b2832a835a7caa309f479e35e015d245ce6812db0ff Jul 15 21:14:12.245915 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.245884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j5bzp" Jul 15 21:14:12.264831 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.264802 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sjtm8" event={"ID":"1f975d04-037c-40d6-b6f6-8283201121b4","Type":"ContainerStarted","Data":"8074d54be2b331c80b296574e0d77a75e99fc8492a89bccb39cc82f653795c8c"} Jul 15 21:14:12.264949 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.264838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sjtm8" event={"ID":"1f975d04-037c-40d6-b6f6-8283201121b4","Type":"ContainerStarted","Data":"f052b844e2befaefde446b2832a835a7caa309f479e35e015d245ce6812db0ff"} Jul 15 21:14:12.283329 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:12.283284 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sjtm8" podStartSLOduration=1.28326751 podStartE2EDuration="1.28326751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:14:12.282282695 +0000 UTC m=+247.523429181" watchObservedRunningTime="2026-07-15 21:14:12.28326751 +0000 UTC m=+247.524413996" Jul 15 21:14:13.795218 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:13.795186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:13.895200 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:13.895170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:20.746770 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:20.746735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:20.846399 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:20.846371 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:22.704137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:22.704098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:34.987998 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:34.987917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:35.083071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.083041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:35.727741 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.727708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:35.885348 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.885315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:35.967789 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.967756 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk"] Jul 15 21:14:35.969855 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.969839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:35.974148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.974123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7nfqq\"" Jul 15 21:14:35.974412 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.974395 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 21:14:35.974688 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.974657 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 21:14:35.974763 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.974698 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 21:14:35.974838 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.974773 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 21:14:35.989213 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.989166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk"] Jul 15 21:14:35.999578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:35.999558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:36.107341 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.107310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:36.141358 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.141337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt485\" (UniqueName: \"kubernetes.io/projected/396ac956-aeb0-4d62-80e6-94c59aa22fbe-kube-api-access-zt485\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.141445 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.141364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.141445 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.141397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-manifests\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.141445 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.141437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-tmp\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.141540 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.141482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.206162 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.206134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:36.242920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.242843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.242920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.242914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt485\" (UniqueName: \"kubernetes.io/projected/396ac956-aeb0-4d62-80e6-94c59aa22fbe-kube-api-access-zt485\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.243027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.242931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.243027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.242964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-manifests\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.243027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.242978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-tmp\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.243470 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.243449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-manifests\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.245128 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.245109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396ac956-aeb0-4d62-80e6-94c59aa22fbe-tmp\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.245384 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.245363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.245444 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.245401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396ac956-aeb0-4d62-80e6-94c59aa22fbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.254079 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.254053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt485\" (UniqueName: \"kubernetes.io/projected/396ac956-aeb0-4d62-80e6-94c59aa22fbe-kube-api-access-zt485\") pod \"opendatahub-operator-controller-manager-5f656749fc-lk5zk\" (UID: \"396ac956-aeb0-4d62-80e6-94c59aa22fbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.280584 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.280562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:36.323287 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.323255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:36.416814 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.416786 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk"] Jul 15 21:14:36.420124 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:14:36.420100 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396ac956_aeb0_4d62_80e6_94c59aa22fbe.slice/crio-9297bc9a4ca50191e255b8898a5d01f34297d0ebdc31c9cce1861816f32d1bb6 WatchSource:0}: Error finding container 9297bc9a4ca50191e255b8898a5d01f34297d0ebdc31c9cce1861816f32d1bb6: Status 404 returned error can't find the container with id 9297bc9a4ca50191e255b8898a5d01f34297d0ebdc31c9cce1861816f32d1bb6 Jul 15 21:14:36.701639 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.701607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:36.805091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:36.805060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:37.362965 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:37.362925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" event={"ID":"396ac956-aeb0-4d62-80e6-94c59aa22fbe","Type":"ContainerStarted","Data":"9297bc9a4ca50191e255b8898a5d01f34297d0ebdc31c9cce1861816f32d1bb6"} Jul 15 21:14:40.374869 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:40.374803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" event={"ID":"396ac956-aeb0-4d62-80e6-94c59aa22fbe","Type":"ContainerStarted","Data":"34b84f310da73f5eadaf3d417b19d7d6ad672bc7ee32ed577a6f8ba4040e67b3"} Jul 15 21:14:40.992252 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:40.992201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:41.092096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:41.092068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:41.380168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:41.380138 2574 generic.go:356] "Generic (PLEG): container finished" podID="396ac956-aeb0-4d62-80e6-94c59aa22fbe" containerID="34b84f310da73f5eadaf3d417b19d7d6ad672bc7ee32ed577a6f8ba4040e67b3" exitCode=0 Jul 15 21:14:41.380531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:41.380196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" event={"ID":"396ac956-aeb0-4d62-80e6-94c59aa22fbe","Type":"ContainerDied","Data":"34b84f310da73f5eadaf3d417b19d7d6ad672bc7ee32ed577a6f8ba4040e67b3"} Jul 15 21:14:42.386186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:42.386156 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" event={"ID":"396ac956-aeb0-4d62-80e6-94c59aa22fbe","Type":"ContainerStarted","Data":"f51ae5dbdbb1c1495cb7b96a67adf230b9ea3365cc1537c2a5175e2d8fba2abd"} Jul 15 21:14:42.386558 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:42.386248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:42.412818 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:42.412773 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" podStartSLOduration=2.902658846 podStartE2EDuration="7.412758861s" totalImagesPullingTime="3.425805228s" totalInitContainerRuntime="1.084294787s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:14:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:14:42.411320779 +0000 UTC m=+277.652467266" watchObservedRunningTime="2026-07-15 21:14:42.412758861 +0000 UTC m=+277.653905344" Jul 15 21:14:53.391458 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:53.391423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f656749fc-lk5zk" Jul 15 21:14:55.125390 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.125342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:55.298829 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.298785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:55.758459 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.758425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:55.852494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.852465 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh"] Jul 15 21:14:55.861977 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.861955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:55.866271 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.866253 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 21:14:55.866499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.866485 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 21:14:55.867315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.867296 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 21:14:55.867402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.867339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9btzr\"" Jul 15 21:14:55.867402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.867370 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 21:14:55.875408 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.875383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh"] Jul 15 21:14:55.903175 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.903144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:55.903315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.903183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7484e6b7-9867-4ab0-9569-97ce4c12597a-tmp\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:55.903315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:55.903253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764dz\" (UniqueName: \"kubernetes.io/projected/7484e6b7-9867-4ab0-9569-97ce4c12597a-kube-api-access-764dz\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.003893 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.003861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.004043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.003903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7484e6b7-9867-4ab0-9569-97ce4c12597a-tmp\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.004043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.003953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764dz\" (UniqueName: \"kubernetes.io/projected/7484e6b7-9867-4ab0-9569-97ce4c12597a-kube-api-access-764dz\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.004043 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:14:56.004019 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 15 21:14:56.004207 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:14:56.004094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs podName:7484e6b7-9867-4ab0-9569-97ce4c12597a nodeName:}" failed. No retries permitted until 2026-07-15 21:14:56.504074004 +0000 UTC m=+291.745220482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs") pod "kube-auth-proxy-59fdd66594-v6rjh" (UID: "7484e6b7-9867-4ab0-9569-97ce4c12597a") : secret "kube-auth-proxy-tls" not found Jul 15 21:14:56.006148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.006128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7484e6b7-9867-4ab0-9569-97ce4c12597a-tmp\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.038191 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.038163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764dz\" (UniqueName: \"kubernetes.io/projected/7484e6b7-9867-4ab0-9569-97ce4c12597a-kube-api-access-764dz\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.085565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.085535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:56.189329 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.189301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:56.317109 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.317034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:56.456448 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.456415 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:56.507879 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.507850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.510092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.510074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7484e6b7-9867-4ab0-9569-97ce4c12597a-tls-certs\") pod \"kube-auth-proxy-59fdd66594-v6rjh\" (UID: \"7484e6b7-9867-4ab0-9569-97ce4c12597a\") " pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.552627 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.552584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:56.771133 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.771054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" Jul 15 21:14:56.921230 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:56.921155 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh"] Jul 15 21:14:56.923362 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:14:56.923336 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7484e6b7_9867_4ab0_9569_97ce4c12597a.slice/crio-543dd3efd2ad6e2d534228d0b845e6df9dfae3e29e55e4a4e2c04072e9a8c223 WatchSource:0}: Error finding container 543dd3efd2ad6e2d534228d0b845e6df9dfae3e29e55e4a4e2c04072e9a8c223: Status 404 returned error can't find the container with id 543dd3efd2ad6e2d534228d0b845e6df9dfae3e29e55e4a4e2c04072e9a8c223 Jul 15 21:14:57.303789 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.303755 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:57.444715 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.444687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" event={"ID":"7484e6b7-9867-4ab0-9569-97ce4c12597a","Type":"ContainerStarted","Data":"543dd3efd2ad6e2d534228d0b845e6df9dfae3e29e55e4a4e2c04072e9a8c223"} Jul 15 21:14:57.732914 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.732838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:57.738260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.735838 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:14:57.741499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.741478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.746077 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.746055 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 21:14:57.746195 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.746055 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 21:14:57.746239 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.746212 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 21:14:57.746754 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.746737 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqvh8\"" Jul 15 21:14:57.746832 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.746817 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 21:14:57.773868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.773849 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:14:57.818527 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818628 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cgc\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818628 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818628 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818769 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818769 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.818832 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.818783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.860738 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.860717 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:57.920001 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.919980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cgc\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920341 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.920777 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.920753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.922262 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.922242 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.922397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.922380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.922455 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.922392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.922494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.922477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.929503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.929485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.932166 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.932146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cgc\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc\") pod \"istiod-openshift-gateway-7574459656-k9l87\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:57.961877 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:57.961851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:58.050558 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:58.050535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:14:58.203825 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:58.203796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:14:58.206609 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:14:58.206579 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010c8990_983b_4e3f_986f_17ea7792eb2e.slice/crio-38844ce73fc2b07c35d6474c19588287150b9658a9f2e6502e568a95fd7fe5ec WatchSource:0}: Error finding container 38844ce73fc2b07c35d6474c19588287150b9658a9f2e6502e568a95fd7fe5ec: Status 404 returned error can't find the container with id 38844ce73fc2b07c35d6474c19588287150b9658a9f2e6502e568a95fd7fe5ec Jul 15 21:14:58.449711 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:58.449606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" event={"ID":"010c8990-983b-4e3f-986f-17ea7792eb2e","Type":"ContainerStarted","Data":"38844ce73fc2b07c35d6474c19588287150b9658a9f2e6502e568a95fd7fe5ec"} Jul 15 21:14:59.203213 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:14:59.203068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:00.873122 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:00.873087 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 21:15:00.873385 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:00.873156 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 21:15:01.465168 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.465129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" event={"ID":"7484e6b7-9867-4ab0-9569-97ce4c12597a","Type":"ContainerStarted","Data":"19eb8bd4fe8f244e59e1b057963ccbea85c73b521a7672cd7c1bb03698851598"} Jul 15 21:15:01.466572 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.466543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" event={"ID":"010c8990-983b-4e3f-986f-17ea7792eb2e","Type":"ContainerStarted","Data":"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9"} Jul 15 21:15:01.466716 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.466703 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:15:01.468324 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.468292 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k9l87 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 21:15:01.468433 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.468346 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 21:15:01.484320 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.484276 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59fdd66594-v6rjh" podStartSLOduration=2.538255684 podStartE2EDuration="6.484263283s" totalImagesPullingTime="3.946007599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:01.482991534 +0000 UTC m=+296.724138043" watchObservedRunningTime="2026-07-15 21:15:01.484263283 +0000 UTC m=+296.725409768" Jul 15 21:15:01.506229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:01.506177 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" podStartSLOduration=1.8425110089999999 podStartE2EDuration="4.506157951s" totalImagesPullingTime="2.663646942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:01.504262355 +0000 UTC m=+296.745408842" watchObservedRunningTime="2026-07-15 21:15:01.506157951 +0000 UTC m=+296.747304439" Jul 15 21:15:02.211359 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.211321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.352005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.351968 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.471385 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.471299 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k9l87 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 21:15:02.471385 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.471344 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 21:15:02.503279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.503245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.776069 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.775979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.966101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:02.966063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:04.852894 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.852861 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff"] Jul 15 21:15:04.856302 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.856282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:04.861435 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.861405 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:15:04.861562 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.861418 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 21:15:04.862955 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.862937 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 21:15:04.863063 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.862968 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 21:15:04.863063 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.863032 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 21:15:04.863170 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.863067 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zwtvz\"" Jul 15 21:15:04.877906 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.877879 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff"] Jul 15 21:15:04.978007 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.977975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:04.978149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.978015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7m7\" (UniqueName: \"kubernetes.io/projected/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-kube-api-access-8c7m7\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:04.978149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.978045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-metrics-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:04.978149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:04.978080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-manager-config\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.078740 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.078710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-metrics-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.078847 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.078749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-manager-config\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.078847 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.078804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.078949 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.078844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7m7\" (UniqueName: \"kubernetes.io/projected/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-kube-api-access-8c7m7\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.079536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.079516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-manager-config\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.081262 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.081224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-metrics-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.081262 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.081251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-cert\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.099773 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.099748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7m7\" (UniqueName: \"kubernetes.io/projected/2f2cdbfb-badc-4465-ac19-fa7fa73b2718-kube-api-access-8c7m7\") pod \"lws-controller-manager-764bf7f9d4-qh6ff\" (UID: \"2f2cdbfb-badc-4465-ac19-fa7fa73b2718\") " pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.146463 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.146387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:15:05.150252 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.150230 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:15:05.154029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.154003 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 21:15:05.169448 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.169429 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zwtvz\"" Jul 15 21:15:05.176536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.176520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:05.343333 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.343291 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff"] Jul 15 21:15:05.346772 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:15:05.346748 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2cdbfb_badc_4465_ac19_fa7fa73b2718.slice/crio-999b733bdcf3b5be3f7e8d753a33210afcfe5e5bc227617440d2c9597bdfb73a WatchSource:0}: Error finding container 999b733bdcf3b5be3f7e8d753a33210afcfe5e5bc227617440d2c9597bdfb73a: Status 404 returned error can't find the container with id 999b733bdcf3b5be3f7e8d753a33210afcfe5e5bc227617440d2c9597bdfb73a Jul 15 21:15:05.471609 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.471531 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:15:05.486203 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:05.486179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" event={"ID":"2f2cdbfb-badc-4465-ac19-fa7fa73b2718","Type":"ContainerStarted","Data":"999b733bdcf3b5be3f7e8d753a33210afcfe5e5bc227617440d2c9597bdfb73a"} Jul 15 21:15:06.935297 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:06.935261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:07.059083 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:07.059020 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:07.146976 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:07.146937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:07.216353 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:07.216270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:08.300818 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.300796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:08.414387 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.414348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:08.505690 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.505588 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" event={"ID":"2f2cdbfb-badc-4465-ac19-fa7fa73b2718","Type":"ContainerStarted","Data":"3af694524df5713c4c890b43893f50322ea8dc561469f31af5efe48916079dc6"} Jul 15 21:15:08.505690 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.505669 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:08.549173 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.549120 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" podStartSLOduration=1.682965056 podStartE2EDuration="4.549103759s" totalImagesPullingTime="2.866138703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:08.545575564 +0000 UTC m=+303.786722052" watchObservedRunningTime="2026-07-15 21:15:08.549103759 +0000 UTC m=+303.790250246" Jul 15 21:15:08.606398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:08.606366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:09.415333 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.415296 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:15:09.415803 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.415526 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" containerID="cri-o://65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9" gracePeriod=30 Jul 15 21:15:09.654308 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.654285 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:15:09.720416 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9cgc\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720416 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720397 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720454 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720497 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720523 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.720560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.720556 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig\") pod \"010c8990-983b-4e3f-986f-17ea7792eb2e\" (UID: \"010c8990-983b-4e3f-986f-17ea7792eb2e\") " Jul 15 21:15:09.721231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.721174 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:15:09.722863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.722827 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts" (OuterVolumeSpecName: "cacerts") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:15:09.722863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.722839 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:15:09.723029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.722983 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs" (OuterVolumeSpecName: "local-certs") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:15:09.723029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.722999 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc" (OuterVolumeSpecName: "kube-api-access-v9cgc") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "kube-api-access-v9cgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:15:09.723153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.723138 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:15:09.723222 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.723198 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "010c8990-983b-4e3f-986f-17ea7792eb2e" (UID: "010c8990-983b-4e3f-986f-17ea7792eb2e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:15:09.821437 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821414 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-dns-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821437 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821434 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-token\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821564 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821444 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-kubeconfig\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821564 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821452 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cgc\" (UniqueName: \"kubernetes.io/projected/010c8990-983b-4e3f-986f-17ea7792eb2e-kube-api-access-v9cgc\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821564 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821460 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010c8990-983b-4e3f-986f-17ea7792eb2e-istio-csr-ca-configmap\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821564 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821469 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010c8990-983b-4e3f-986f-17ea7792eb2e-local-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:09.821564 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:09.821477 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010c8990-983b-4e3f-986f-17ea7792eb2e-cacerts\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:15:10.514852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.514817 2574 generic.go:356] "Generic (PLEG): container finished" podID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerID="65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9" exitCode=0 Jul 15 21:15:10.515272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.514873 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" Jul 15 21:15:10.515272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.514897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" event={"ID":"010c8990-983b-4e3f-986f-17ea7792eb2e","Type":"ContainerDied","Data":"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9"} Jul 15 21:15:10.515272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.514946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9l87" event={"ID":"010c8990-983b-4e3f-986f-17ea7792eb2e","Type":"ContainerDied","Data":"38844ce73fc2b07c35d6474c19588287150b9658a9f2e6502e568a95fd7fe5ec"} Jul 15 21:15:10.515272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.514966 2574 scope.go:117] "RemoveContainer" containerID="65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9" Jul 15 21:15:10.523611 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.523596 2574 scope.go:117] "RemoveContainer" containerID="65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9" Jul 15 21:15:10.523870 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:15:10.523851 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9\": container with ID starting with 65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9 not found: ID does not exist" containerID="65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9" Jul 15 21:15:10.523925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.523877 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9"} err="failed to get container status \"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9\": rpc error: code = NotFound desc = could not find container \"65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9\": container with ID starting with 65714d78b78388ba6f50abe03cb33fbfdc82291e598fcef78f2863c7768495a9 not found: ID does not exist" Jul 15 21:15:10.537051 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.537024 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:15:10.540145 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:10.540123 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9l87"] Jul 15 21:15:11.292593 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:11.292559 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" path="/var/lib/kubelet/pods/010c8990-983b-4e3f-986f-17ea7792eb2e/volumes" Jul 15 21:15:19.511824 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:19.511787 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-764bf7f9d4-qh6ff" Jul 15 21:15:44.748629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:44.748595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:56.664651 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:56.664615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:56.738973 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:56.738938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:56.840693 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:15:56.840655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:03.621519 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:03.621487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:22.503907 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:22.503872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:25.535638 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:25.535605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:26.165071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:26.165030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:30.338588 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:30.338553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:32.605901 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:32.605866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:32.740936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:32.740902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:35.343028 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:35.342998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:35.569602 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:35.569571 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:37.276042 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.276006 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr"] Jul 15 21:16:37.276400 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.276335 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" Jul 15 21:16:37.276400 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.276347 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" Jul 15 21:16:37.276467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.276424 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="010c8990-983b-4e3f-986f-17ea7792eb2e" containerName="discovery" Jul 15 21:16:37.279317 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.279300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:37.282276 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.282258 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 15 21:16:37.282356 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.282289 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 21:16:37.283644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.283628 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 21:16:37.283773 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.283630 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cr6c2\"" Jul 15 21:16:37.292289 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.292270 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr"] Jul 15 21:16:37.458070 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.458039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8k5\" (UniqueName: \"kubernetes.io/projected/59271228-bf3a-4908-94de-2c05b8c05bf3-kube-api-access-gd8k5\") pod \"dns-operator-controller-manager-864dd78c4c-99pwr\" (UID: \"59271228-bf3a-4908-94de-2c05b8c05bf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:37.558489 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.558456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8k5\" (UniqueName: \"kubernetes.io/projected/59271228-bf3a-4908-94de-2c05b8c05bf3-kube-api-access-gd8k5\") pod \"dns-operator-controller-manager-864dd78c4c-99pwr\" (UID: \"59271228-bf3a-4908-94de-2c05b8c05bf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:37.567621 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.567594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8k5\" (UniqueName: \"kubernetes.io/projected/59271228-bf3a-4908-94de-2c05b8c05bf3-kube-api-access-gd8k5\") pod \"dns-operator-controller-manager-864dd78c4c-99pwr\" (UID: \"59271228-bf3a-4908-94de-2c05b8c05bf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:37.589468 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.589440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:37.724043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.724021 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr"] Jul 15 21:16:37.726523 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:16:37.726495 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59271228_bf3a_4908_94de_2c05b8c05bf3.slice/crio-1b20099ffa66eec63ee825b6924d1bc80f1598b3dab5034ad666dd2039a26fb2 WatchSource:0}: Error finding container 1b20099ffa66eec63ee825b6924d1bc80f1598b3dab5034ad666dd2039a26fb2: Status 404 returned error can't find the container with id 1b20099ffa66eec63ee825b6924d1bc80f1598b3dab5034ad666dd2039a26fb2 Jul 15 21:16:37.728653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.728629 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:16:37.857586 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.857509 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" event={"ID":"59271228-bf3a-4908-94de-2c05b8c05bf3","Type":"ContainerStarted","Data":"1b20099ffa66eec63ee825b6924d1bc80f1598b3dab5034ad666dd2039a26fb2"} Jul 15 21:16:37.932766 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:37.932734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:38.611002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:38.610968 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:39.777967 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.777941 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc"] Jul 15 21:16:39.782301 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.782282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:39.786050 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.786033 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-996l8\"" Jul 15 21:16:39.792932 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.792909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc"] Jul 15 21:16:39.867183 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.867150 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" event={"ID":"59271228-bf3a-4908-94de-2c05b8c05bf3","Type":"ContainerStarted","Data":"0853fe16e5e1d14e454411bcdd80bf5e5b79dd6c51befed21db2149b85f8d151"} Jul 15 21:16:39.867328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.867284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:39.885829 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.885787 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" podStartSLOduration=0.863715729 podStartE2EDuration="2.885773354s" totalImagesPullingTime="2.022057625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:39.885374453 +0000 UTC m=+395.126520974" watchObservedRunningTime="2026-07-15 21:16:39.885773354 +0000 UTC m=+395.126919841" Jul 15 21:16:39.982086 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:39.982048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6xb4\" (UniqueName: \"kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4\") pod \"limitador-operator-controller-manager-859d7b78bd-29jhc\" (UID: \"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:40.082995 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:40.082953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xb4\" (UniqueName: \"kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4\") pod \"limitador-operator-controller-manager-859d7b78bd-29jhc\" (UID: \"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:40.091874 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:40.091840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xb4\" (UniqueName: \"kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4\") pod \"limitador-operator-controller-manager-859d7b78bd-29jhc\" (UID: \"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:40.101603 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:40.101578 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:40.231551 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:40.231520 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc"] Jul 15 21:16:40.234669 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:16:40.234641 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34dd8e3_b686_4a32_9ea7_b7f8a51eb2d5.slice/crio-8bdf1ccb1a4d36d89a3635f92adf6d72d112b18bbd386f4f71ba92d296c0ae52 WatchSource:0}: Error finding container 8bdf1ccb1a4d36d89a3635f92adf6d72d112b18bbd386f4f71ba92d296c0ae52: Status 404 returned error can't find the container with id 8bdf1ccb1a4d36d89a3635f92adf6d72d112b18bbd386f4f71ba92d296c0ae52 Jul 15 21:16:40.872928 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:40.872891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" event={"ID":"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5","Type":"ContainerStarted","Data":"8bdf1ccb1a4d36d89a3635f92adf6d72d112b18bbd386f4f71ba92d296c0ae52"} Jul 15 21:16:41.016536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:41.016503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:41.729926 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:41.729889 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:41.785127 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:41.785096 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:41.985916 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:41.985838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:42.061724 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.061659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:42.162917 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.162870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:42.263362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.263336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:42.885363 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.885326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" event={"ID":"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5","Type":"ContainerStarted","Data":"204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852"} Jul 15 21:16:42.885533 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.885398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:42.906236 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:42.906181 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" podStartSLOduration=1.931524134 podStartE2EDuration="3.906163124s" totalImagesPullingTime="1.97463899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:42.904893875 +0000 UTC m=+398.146040423" watchObservedRunningTime="2026-07-15 21:16:42.906163124 +0000 UTC m=+398.147309611" Jul 15 21:16:43.151867 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:43.151786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:43.212744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:43.212705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:45.736532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:45.736487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:45.870471 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:45.870437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:50.874951 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:50.874923 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-99pwr" Jul 15 21:16:53.036324 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.036289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:53.127874 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.127843 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:53.228205 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.228178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:53.339216 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.339120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:53.437762 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.437731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:53.657896 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.657819 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc"] Jul 15 21:16:53.658105 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.658078 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" containerName="manager" containerID="cri-o://204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852" gracePeriod=2 Jul 15 21:16:53.659772 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.659745 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.673380 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.673354 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc"] Jul 15 21:16:53.675280 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.675249 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-29jhc\": pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.677708 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.677657 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.687503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.687483 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss"] Jul 15 21:16:53.687791 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.687779 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" containerName="manager" Jul 15 21:16:53.687837 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.687794 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" containerName="manager" Jul 15 21:16:53.687870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.687854 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" containerName="manager" Jul 15 21:16:53.690756 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.690732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:53.697814 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.697775 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.705008 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.704987 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss"] Jul 15 21:16:53.798825 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.798792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslm7\" (UniqueName: \"kubernetes.io/projected/6926776c-3988-47cd-b382-d847cbb41a9a-kube-api-access-fslm7\") pod \"limitador-operator-controller-manager-859d7b78bd-4tzss\" (UID: \"6926776c-3988-47cd-b382-d847cbb41a9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:53.897375 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.897350 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.899480 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.899462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslm7\" (UniqueName: \"kubernetes.io/projected/6926776c-3988-47cd-b382-d847cbb41a9a-kube-api-access-fslm7\") pod \"limitador-operator-controller-manager-859d7b78bd-4tzss\" (UID: \"6926776c-3988-47cd-b382-d847cbb41a9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:53.899730 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.899708 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.911906 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.911837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslm7\" (UniqueName: \"kubernetes.io/projected/6926776c-3988-47cd-b382-d847cbb41a9a-kube-api-access-fslm7\") pod \"limitador-operator-controller-manager-859d7b78bd-4tzss\" (UID: \"6926776c-3988-47cd-b382-d847cbb41a9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:53.937103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.937077 2574 generic.go:356] "Generic (PLEG): container finished" podID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" containerID="204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852" exitCode=0 Jul 15 21:16:53.937198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.937118 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.937198 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.937129 2574 scope.go:117] "RemoveContainer" containerID="204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852" Jul 15 21:16:53.939902 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.939873 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:53.944337 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.944322 2574 scope.go:117] "RemoveContainer" containerID="204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852" Jul 15 21:16:53.944569 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:53.944550 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852\": container with ID starting with 204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852 not found: ID does not exist" containerID="204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852" Jul 15 21:16:53.944618 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:53.944579 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852"} err="failed to get container status \"204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852\": rpc error: code = NotFound desc = could not find container \"204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852\": container with ID starting with 204168afeea7777f9e1bdb7bf1d2e4bde630de92aed76517649ef7d60cba4852 not found: ID does not exist" Jul 15 21:16:54.000135 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.000114 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6xb4\" (UniqueName: \"kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4\") pod \"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\" (UID: \"b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5\") " Jul 15 21:16:54.002038 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.002020 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4" (OuterVolumeSpecName: "kube-api-access-c6xb4") pod "b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" (UID: "b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5"). InnerVolumeSpecName "kube-api-access-c6xb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:16:54.031210 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.031189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:54.100923 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.100896 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6xb4\" (UniqueName: \"kubernetes.io/projected/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5-kube-api-access-c6xb4\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:16:54.165309 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.165227 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss"] Jul 15 21:16:54.168618 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:16:54.168590 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6926776c_3988_47cd_b382_d847cbb41a9a.slice/crio-a32d1ec8ed60c28c5dc9cf433a6c52dc32f880591d1207fad9a698a26b8aa941 WatchSource:0}: Error finding container a32d1ec8ed60c28c5dc9cf433a6c52dc32f880591d1207fad9a698a26b8aa941: Status 404 returned error can't find the container with id a32d1ec8ed60c28c5dc9cf433a6c52dc32f880591d1207fad9a698a26b8aa941 Jul 15 21:16:54.252295 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:54.252262 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:54.944047 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.944006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" event={"ID":"6926776c-3988-47cd-b382-d847cbb41a9a","Type":"ContainerStarted","Data":"fcfb852d3118ec7e1d6fb855243e65606ed8ec1248fabe289edb2d68783183ea"} Jul 15 21:16:54.944253 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.944071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" event={"ID":"6926776c-3988-47cd-b382-d847cbb41a9a","Type":"ContainerStarted","Data":"a32d1ec8ed60c28c5dc9cf433a6c52dc32f880591d1207fad9a698a26b8aa941"} Jul 15 21:16:54.944253 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.944100 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:16:54.948007 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:54.947978 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:54.981629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:54.981584 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" podStartSLOduration=1.981570274 podStartE2EDuration="1.981570274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:54.979401703 +0000 UTC m=+410.220548202" watchObservedRunningTime="2026-07-15 21:16:54.981570274 +0000 UTC m=+410.222716761" Jul 15 21:16:55.291237 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:16:55.291191 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-29jhc\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29jhc" Jul 15 21:16:55.292477 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:55.292448 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5" path="/var/lib/kubelet/pods/b34dd8e3-b686-4a32-9ea7-b7f8a51eb2d5/volumes" Jul 15 21:16:56.954365 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:56.954330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:57.056637 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:16:57.056606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:00.746841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:00.746795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:00.802801 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:00.802764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:00.945415 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:00.945383 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:01.008348 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:01.008277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:03.491819 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:03.491758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:03.648922 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:03.648887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:05.950118 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:05.950091 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tzss" Jul 15 21:17:09.981128 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:09.981089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:10.078820 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:10.078761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:14.725765 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:14.725730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:14.831962 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:14.831929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:18.548239 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:18.548206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:18.615804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:18.615759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:19.139865 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:19.139837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:19.196440 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:19.196409 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:22.490856 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:22.490825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:22.681515 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:22.681487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:22.936266 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:22.936238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:23.005264 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:23.005231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:24.183850 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.183817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:24.304924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.304882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:24.367533 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.367502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:24.571770 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.571735 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:17:24.579475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.579455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.582190 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.582168 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 21:17:24.582303 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.582250 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2ggjl\"" Jul 15 21:17:24.586437 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.586075 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:17:24.621512 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.621480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:17:24.739991 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.739953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxpc\" (UniqueName: \"kubernetes.io/projected/2adf7ba2-7cd0-4811-b78e-7911c879f485-kube-api-access-rzxpc\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.739991 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.739988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2adf7ba2-7cd0-4811-b78e-7911c879f485-config-file\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.841293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.841210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxpc\" (UniqueName: \"kubernetes.io/projected/2adf7ba2-7cd0-4811-b78e-7911c879f485-kube-api-access-rzxpc\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.841293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.841248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2adf7ba2-7cd0-4811-b78e-7911c879f485-config-file\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.841891 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.841872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2adf7ba2-7cd0-4811-b78e-7911c879f485-config-file\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.849638 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.849602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxpc\" (UniqueName: \"kubernetes.io/projected/2adf7ba2-7cd0-4811-b78e-7911c879f485-kube-api-access-rzxpc\") pod \"limitador-limitador-54d6d988d5-ghvfw\" (UID: \"2adf7ba2-7cd0-4811-b78e-7911c879f485\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:24.890627 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:24.890601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:25.026144 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.026122 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:17:25.027927 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:17:25.027895 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adf7ba2_7cd0_4811_b78e_7911c879f485.slice/crio-4e7859823d2202279a5136bf3867c533a1a0d5ea203c263dd45c7fe94aac342d WatchSource:0}: Error finding container 4e7859823d2202279a5136bf3867c533a1a0d5ea203c263dd45c7fe94aac342d: Status 404 returned error can't find the container with id 4e7859823d2202279a5136bf3867c533a1a0d5ea203c263dd45c7fe94aac342d Jul 15 21:17:25.037293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.037261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:25.074945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.074921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" event={"ID":"2adf7ba2-7cd0-4811-b78e-7911c879f485","Type":"ContainerStarted","Data":"4e7859823d2202279a5136bf3867c533a1a0d5ea203c263dd45c7fe94aac342d"} Jul 15 21:17:25.124618 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.124549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:25.203990 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.203959 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:17:25.208656 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.208640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:17:25.211635 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.211618 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rf2ts\"" Jul 15 21:17:25.217447 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.217426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:17:25.346687 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.346644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbzl\" (UniqueName: \"kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl\") pod \"authorino-75d556bd77-jqwkx\" (UID: \"6801ff98-9c63-4368-8914-0752eb7da880\") " pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:17:25.447826 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.447736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbzl\" (UniqueName: \"kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl\") pod \"authorino-75d556bd77-jqwkx\" (UID: \"6801ff98-9c63-4368-8914-0752eb7da880\") " pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:17:25.457743 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.457704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbzl\" (UniqueName: \"kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl\") pod \"authorino-75d556bd77-jqwkx\" (UID: \"6801ff98-9c63-4368-8914-0752eb7da880\") " pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:17:25.518154 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.518117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:17:25.684537 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:25.684507 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:17:25.688324 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:17:25.688296 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6801ff98_9c63_4368_8914_0752eb7da880.slice/crio-63f8a7833cd9cc8b7102faabfa458e68e03626d9ae48048bdc87524e5ff0151d WatchSource:0}: Error finding container 63f8a7833cd9cc8b7102faabfa458e68e03626d9ae48048bdc87524e5ff0151d: Status 404 returned error can't find the container with id 63f8a7833cd9cc8b7102faabfa458e68e03626d9ae48048bdc87524e5ff0151d Jul 15 21:17:26.085841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:26.084149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jqwkx" event={"ID":"6801ff98-9c63-4368-8914-0752eb7da880","Type":"ContainerStarted","Data":"63f8a7833cd9cc8b7102faabfa458e68e03626d9ae48048bdc87524e5ff0151d"} Jul 15 21:17:28.096753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:28.096713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" event={"ID":"2adf7ba2-7cd0-4811-b78e-7911c879f485","Type":"ContainerStarted","Data":"df573972a3a39f2a26bb23320748b059d1e264a3d6cd40723cddcee760441380"} Jul 15 21:17:28.097298 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:28.096897 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:28.116082 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:28.116022 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" podStartSLOduration=1.445085904 podStartE2EDuration="4.116001101s" totalImagesPullingTime="2.670915197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:17:28.114609338 +0000 UTC m=+443.355755847" watchObservedRunningTime="2026-07-15 21:17:28.116001101 +0000 UTC m=+443.357147589" Jul 15 21:17:28.987738 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:28.987698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:29.087127 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:29.087091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:30.106786 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:30.106736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jqwkx" event={"ID":"6801ff98-9c63-4368-8914-0752eb7da880","Type":"ContainerStarted","Data":"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a"} Jul 15 21:17:30.123177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:30.122997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jqwkx" podStartSLOduration=1.257569207 podStartE2EDuration="5.122980669s" totalImagesPullingTime="3.865411462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:17:30.122721825 +0000 UTC m=+445.363868312" watchObservedRunningTime="2026-07-15 21:17:30.122980669 +0000 UTC m=+445.364127156" Jul 15 21:17:33.776074 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:33.776045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:33.872424 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:33.872398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:39.101306 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:39.101275 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ghvfw" Jul 15 21:17:45.368651 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:45.368615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:45.468148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:45.468118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:45.796929 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:45.796894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:58.202024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:58.201993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:58.305808 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:17:58.305780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:00.341088 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.341054 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:18:00.341454 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.341237 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jqwkx" podUID="6801ff98-9c63-4368-8914-0752eb7da880" containerName="authorino" containerID="cri-o://9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a" gracePeriod=30 Jul 15 21:18:00.589607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.589585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:18:00.624488 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.624424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlbzl\" (UniqueName: \"kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl\") pod \"6801ff98-9c63-4368-8914-0752eb7da880\" (UID: \"6801ff98-9c63-4368-8914-0752eb7da880\") " Jul 15 21:18:00.626412 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.626389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl" (OuterVolumeSpecName: "kube-api-access-hlbzl") pod "6801ff98-9c63-4368-8914-0752eb7da880" (UID: "6801ff98-9c63-4368-8914-0752eb7da880"). InnerVolumeSpecName "kube-api-access-hlbzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:18:00.725027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:00.724993 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbzl\" (UniqueName: \"kubernetes.io/projected/6801ff98-9c63-4368-8914-0752eb7da880-kube-api-access-hlbzl\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:18:01.240987 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.240948 2574 generic.go:356] "Generic (PLEG): container finished" podID="6801ff98-9c63-4368-8914-0752eb7da880" containerID="9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a" exitCode=0 Jul 15 21:18:01.241148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.241014 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jqwkx" Jul 15 21:18:01.241148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.241030 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jqwkx" event={"ID":"6801ff98-9c63-4368-8914-0752eb7da880","Type":"ContainerDied","Data":"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a"} Jul 15 21:18:01.241148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.241072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jqwkx" event={"ID":"6801ff98-9c63-4368-8914-0752eb7da880","Type":"ContainerDied","Data":"63f8a7833cd9cc8b7102faabfa458e68e03626d9ae48048bdc87524e5ff0151d"} Jul 15 21:18:01.241148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.241089 2574 scope.go:117] "RemoveContainer" containerID="9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a" Jul 15 21:18:01.249613 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.249597 2574 scope.go:117] "RemoveContainer" containerID="9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a" Jul 15 21:18:01.249871 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:18:01.249852 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a\": container with ID starting with 9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a not found: ID does not exist" containerID="9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a" Jul 15 21:18:01.249922 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.249877 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a"} err="failed to get container status \"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a\": rpc error: code = NotFound desc = could not find container \"9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a\": container with ID starting with 9a4c2a8b223cd54e4b5afb625514b231ceda2674e69452d06e674ec76b57119a not found: ID does not exist" Jul 15 21:18:01.268129 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.268105 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:18:01.272589 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.272565 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jqwkx"] Jul 15 21:18:01.292275 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:01.292249 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6801ff98-9c63-4368-8914-0752eb7da880" path="/var/lib/kubelet/pods/6801ff98-9c63-4368-8914-0752eb7da880/volumes" Jul 15 21:18:05.278498 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:05.278463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:05.395624 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:05.395593 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:05.460859 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:05.460787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:05.789490 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:05.789461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:09.417061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:09.417014 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:09.527963 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:09.527928 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:24.379711 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:24.379451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:30.947820 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:30.947783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:35.784418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:35.784385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:35.891194 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:35.891163 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:36.169508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:36.169412 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:38.929857 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:38.929821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:39.032127 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:39.032094 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:40.226089 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:40.226053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:40.325088 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:40.325058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:40.746771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:40.746735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:40.843505 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:40.843474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:41.248669 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:41.248639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:41.344022 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:41.343997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:41.519719 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:41.519605 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:18:51.073383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:51.073350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:51.175958 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:18:51.175929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:07.266868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:07.266783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:07.716812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:07.716776 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:19:13.355023 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:13.354989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:14.415113 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:14.415084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:19:28.549868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:28.549837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:29.631638 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:29.631603 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:19:49.067887 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:49.067853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:50.629377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:50.629345 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:19:54.912056 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.912017 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr"] Jul 15 21:19:54.912499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.912338 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6801ff98-9c63-4368-8914-0752eb7da880" containerName="authorino" Jul 15 21:19:54.912499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.912350 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6801ff98-9c63-4368-8914-0752eb7da880" containerName="authorino" Jul 15 21:19:54.912499 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.912396 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6801ff98-9c63-4368-8914-0752eb7da880" containerName="authorino" Jul 15 21:19:54.915328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.915308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.918235 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.918215 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 21:19:54.918352 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.918248 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 21:19:54.918352 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.918289 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 21:19:54.918352 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.918248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jz2qs\"" Jul 15 21:19:54.926478 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.926458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr"] Jul 15 21:19:54.942170 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.942151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:54.970221 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.970325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.970385 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.970438 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.970501 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpz7n\" (UniqueName: \"kubernetes.io/projected/a677a8dd-bb7b-402b-bb6f-59c1e444866f-kube-api-access-cpz7n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:54.970573 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:54.970557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.071854 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.071820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.071997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.071878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.071997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.071910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.071997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.071939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.071997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.071975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.072203 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.072017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpz7n\" (UniqueName: \"kubernetes.io/projected/a677a8dd-bb7b-402b-bb6f-59c1e444866f-kube-api-access-cpz7n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.072347 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.072321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.072418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.072356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.072418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.072386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.074146 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.074119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a677a8dd-bb7b-402b-bb6f-59c1e444866f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.074373 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.074355 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a677a8dd-bb7b-402b-bb6f-59c1e444866f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.079979 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.079949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpz7n\" (UniqueName: \"kubernetes.io/projected/a677a8dd-bb7b-402b-bb6f-59c1e444866f-kube-api-access-cpz7n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kncdr\" (UID: \"a677a8dd-bb7b-402b-bb6f-59c1e444866f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.225782 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.225694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:19:55.357899 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.357873 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr"] Jul 15 21:19:55.360423 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:19:55.360394 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda677a8dd_bb7b_402b_bb6f_59c1e444866f.slice/crio-833ef145074cdaead54e37413f6c824b914bd3732adb0285e2bf229a27716b10 WatchSource:0}: Error finding container 833ef145074cdaead54e37413f6c824b914bd3732adb0285e2bf229a27716b10: Status 404 returned error can't find the container with id 833ef145074cdaead54e37413f6c824b914bd3732adb0285e2bf229a27716b10 Jul 15 21:19:55.735227 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:55.735190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" event={"ID":"a677a8dd-bb7b-402b-bb6f-59c1e444866f","Type":"ContainerStarted","Data":"833ef145074cdaead54e37413f6c824b914bd3732adb0285e2bf229a27716b10"} Jul 15 21:19:56.741012 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:56.740929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" event={"ID":"a677a8dd-bb7b-402b-bb6f-59c1e444866f","Type":"ContainerStarted","Data":"071d1857d815d45d848dc131f545f73e14e471b2fa6c486c2e7ed077900a5597"} Jul 15 21:19:56.762813 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:56.762760 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" podStartSLOduration=1.768657973 podStartE2EDuration="2.762745836s" totalImagesPullingTime="994.087863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:19:56.760049829 +0000 UTC m=+592.001196318" watchObservedRunningTime="2026-07-15 21:19:56.762745836 +0000 UTC m=+592.003892321" Jul 15 21:19:58.214117 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:19:58.214084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:20:02.396335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:02.396286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:03.850094 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:03.850059 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:05.178653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.178628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:20:05.179349 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.179335 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:20:05.226143 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.226112 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:20:05.226307 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.226254 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:20:05.239081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.239058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:20:05.790288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:05.790260 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kncdr" Jul 15 21:20:16.020034 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.020003 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5"] Jul 15 21:20:16.023243 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.023224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.026209 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.026190 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 21:20:16.038861 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.038841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5"] Jul 15 21:20:16.063002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.062979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:16.149557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdce663-a6c9-4a7d-904d-e283d4657127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.149771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.149771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.149771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.149771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.149771 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.149745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgv8\" (UniqueName: \"kubernetes.io/projected/1cdce663-a6c9-4a7d-904d-e283d4657127-kube-api-access-6tgv8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.250954 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.250919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.250954 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.250956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251196 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.250976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251196 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.250994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251196 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.251019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgv8\" (UniqueName: \"kubernetes.io/projected/1cdce663-a6c9-4a7d-904d-e283d4657127-kube-api-access-6tgv8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251196 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.251071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdce663-a6c9-4a7d-904d-e283d4657127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.251348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.251420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.251486 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.251462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.253252 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.253229 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdce663-a6c9-4a7d-904d-e283d4657127-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.253489 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.253471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdce663-a6c9-4a7d-904d-e283d4657127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.259445 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.259423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgv8\" (UniqueName: \"kubernetes.io/projected/1cdce663-a6c9-4a7d-904d-e283d4657127-kube-api-access-6tgv8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5\" (UID: \"1cdce663-a6c9-4a7d-904d-e283d4657127\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.333215 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.333190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:16.463697 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.463567 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5"] Jul 15 21:20:16.465933 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:20:16.465900 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cdce663_a6c9_4a7d_904d_e283d4657127.slice/crio-4f3edb2b079b3dfa9e8c8d5beafdc3e8ea2a06b64fc4998791c72e6884449fb7 WatchSource:0}: Error finding container 4f3edb2b079b3dfa9e8c8d5beafdc3e8ea2a06b64fc4998791c72e6884449fb7: Status 404 returned error can't find the container with id 4f3edb2b079b3dfa9e8c8d5beafdc3e8ea2a06b64fc4998791c72e6884449fb7 Jul 15 21:20:16.831472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.826209 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" event={"ID":"1cdce663-a6c9-4a7d-904d-e283d4657127","Type":"ContainerStarted","Data":"5f866cb7c08930d22b24b2a5ebba03627dbe8d1eeafb1762205ab45bcece271a"} Jul 15 21:20:16.831472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.826626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" event={"ID":"1cdce663-a6c9-4a7d-904d-e283d4657127","Type":"ContainerStarted","Data":"4f3edb2b079b3dfa9e8c8d5beafdc3e8ea2a06b64fc4998791c72e6884449fb7"} Jul 15 21:20:16.848988 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:16.848943 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" podStartSLOduration=1.657343252 podStartE2EDuration="1.84892809s" totalImagesPullingTime="191.584838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:20:16.848111517 +0000 UTC m=+612.089258004" watchObservedRunningTime="2026-07-15 21:20:16.84892809 +0000 UTC m=+612.090074578" Jul 15 21:20:18.728663 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:18.728629 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:20:26.333829 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:26.333788 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:26.334260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:26.333843 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:26.346223 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:26.346199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:26.883981 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:26.883951 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5" Jul 15 21:20:41.265102 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:41.265069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:41.320006 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:20:41.319976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:56.543967 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.543934 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76f848cf5-qdk6j"] Jul 15 21:21:56.546541 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.546517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.550757 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.550725 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rf2ts\"" Jul 15 21:21:56.550757 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.550725 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 21:21:56.562051 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.562028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76f848cf5-qdk6j"] Jul 15 21:21:56.642557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.642518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hn4\" (UniqueName: \"kubernetes.io/projected/14965a65-b377-4d8d-9a6e-98b4a44a8a78-kube-api-access-77hn4\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.642743 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.642586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14965a65-b377-4d8d-9a6e-98b4a44a8a78-tls-cert\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.743368 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.743324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14965a65-b377-4d8d-9a6e-98b4a44a8a78-tls-cert\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.743525 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.743405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hn4\" (UniqueName: \"kubernetes.io/projected/14965a65-b377-4d8d-9a6e-98b4a44a8a78-kube-api-access-77hn4\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.745880 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.745859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14965a65-b377-4d8d-9a6e-98b4a44a8a78-tls-cert\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.753149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.753111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hn4\" (UniqueName: \"kubernetes.io/projected/14965a65-b377-4d8d-9a6e-98b4a44a8a78-kube-api-access-77hn4\") pod \"authorino-76f848cf5-qdk6j\" (UID: \"14965a65-b377-4d8d-9a6e-98b4a44a8a78\") " pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.827037 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.826949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:56.855420 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.855397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f848cf5-qdk6j" Jul 15 21:21:56.922823 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.922753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:56.994715 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.994692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76f848cf5-qdk6j"] Jul 15 21:21:56.996754 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:21:56.996719 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14965a65_b377_4d8d_9a6e_98b4a44a8a78.slice/crio-6fd393367c12553fa6cb32ea03d8ad0a99342b7afddda47aa639e928171e7abf WatchSource:0}: Error finding container 6fd393367c12553fa6cb32ea03d8ad0a99342b7afddda47aa639e928171e7abf: Status 404 returned error can't find the container with id 6fd393367c12553fa6cb32ea03d8ad0a99342b7afddda47aa639e928171e7abf Jul 15 21:21:56.998149 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:56.998129 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:21:57.257991 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:57.257908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f848cf5-qdk6j" event={"ID":"14965a65-b377-4d8d-9a6e-98b4a44a8a78","Type":"ContainerStarted","Data":"6fd393367c12553fa6cb32ea03d8ad0a99342b7afddda47aa639e928171e7abf"} Jul 15 21:21:58.059135 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:58.059091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:58.152629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:58.152566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:58.265449 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:58.265407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f848cf5-qdk6j" event={"ID":"14965a65-b377-4d8d-9a6e-98b4a44a8a78","Type":"ContainerStarted","Data":"f9e86938d8abdc151f2b8bf7d708af5040512c5f66100e2607a11e9f4fd248ec"} Jul 15 21:21:58.303235 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:21:58.303171 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76f848cf5-qdk6j" podStartSLOduration=1.809699017 podStartE2EDuration="2.303149974s" totalImagesPullingTime="493.450957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:21:58.302154629 +0000 UTC m=+713.543301116" watchObservedRunningTime="2026-07-15 21:21:58.303149974 +0000 UTC m=+713.544296463" Jul 15 21:22:26.130336 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:22:26.130306 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:22:35.912653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:22:35.912617 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:22:46.026379 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:22:46.026342 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:22:54.723247 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:22:54.723212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:23:05.418030 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:05.417995 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:23:08.315532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:08.315498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:10.051066 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:10.051024 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:34.624494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:34.624401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:23:46.925816 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:46.925784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:23:51.630934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:51.630898 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:51.729036 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:51.729003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:51.852741 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:51.852706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:57.760769 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:57.760737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:57.825875 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:57.825846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:57.920004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:57.919970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:58.016815 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:58.016733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:58.130602 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:23:58.130573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:03.964894 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:03.964849 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:04.065791 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:04.065756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:20.919990 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:20.919958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:24:31.920558 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:31.920518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:24:33.081640 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:33.081605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:37.848326 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:37.848292 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:37.949574 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:37.949538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:43.021546 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:43.021511 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:24:52.447797 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:24:52.447762 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:25:03.314328 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:03.314232 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:25:05.229606 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:05.229576 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:25:05.230078 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:05.229754 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:25:12.516976 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:12.516945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:25:24.620411 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:24.620374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:25:34.021388 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:34.021355 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:25:36.488030 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:36.487992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:25:55.058429 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:55.058390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:25:57.114067 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:25:57.114031 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:26:07.415582 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:26:07.415549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:26:32.620302 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:26:32.620209 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:26:42.822428 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:26:42.822397 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:27:08.931123 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:27:08.931085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:27:18.624531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:27:18.624491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:27:27.912516 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:27:27.912483 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:27:37.414178 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:27:37.414145 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:28:43.753494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:28:43.753461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:28:43.838703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:28:43.838655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:29:26.123272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:29:26.123234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:29:37.815270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:29:37.815179 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:29:46.515394 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:29:46.515358 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:29:57.832701 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:29:57.832185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:05.263951 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:05.263922 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:30:05.264358 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:05.263922 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:30:09.114948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:09.114907 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:20.223053 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:20.223017 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:29.130980 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:29.130932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:39.913567 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:39.913528 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:51.213888 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:51.213847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:30:59.919029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:30:59.918929 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:31:10.414592 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:31:10.414551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:31:35.109290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:31:35.109252 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:31:42.529760 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:31:42.529717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:31:52.016432 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:31:52.016393 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:32:02.719736 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:02.719692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:32:09.316933 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:09.316894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:32:21.032599 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:21.032559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:27.220650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:27.220561 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:32:39.027165 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:39.027129 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:32:48.324262 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:32:48.324223 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:01.919138 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:01.919099 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:10.618290 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:10.618250 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:25.024042 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:25.024004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:34.423916 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:34.423879 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:44.024594 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:44.024553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:33:54.422078 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:33:54.422040 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:07.415250 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:07.415170 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:14.019989 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:14.019951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:30.613169 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:30.613128 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:41.817895 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:41.817854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:50.814152 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:50.814110 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:34:54.598275 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:34:54.598240 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:01.056923 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:01.056881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:02.481246 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:02.481197 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:03.618371 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:03.618336 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:35:05.082546 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:05.082498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:05.290325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:05.290292 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:35:05.291114 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:05.291085 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:35:09.337908 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:09.337872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:13.388729 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:13.383659 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:35:14.436623 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:14.436584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:15.609105 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:15.609068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.488071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:17.488040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.585393 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:17.585337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.946756 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:17.946715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:22.621504 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:22.621454 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:35:23.779361 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:23.779327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:23.988960 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:23.988922 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:24.142080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:24.142047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:24.242426 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:24.242387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:24.343311 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:24.343230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:24.487244 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:24.487210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:25.034541 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:25.034508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:25.134965 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:25.134926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:28.182199 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:28.182175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:30.389034 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:30.388999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:30.510349 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:30.510311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:32.477277 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:32.477221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.704650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:36.704605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.797363 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:36.797322 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.922479 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:36.922448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.010165 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.010072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.233876 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.233838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.381191 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.381154 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.483743 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.483710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.575095 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.575065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.768005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:37.767912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:38.054352 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:38.054316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:38.160061 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:38.160028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:40.315967 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:40.315933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:35:40.341812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:40.341777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:43.003448 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:43.003414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:43.103249 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:43.103211 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:44.597391 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:44.597350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.356477 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.356442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.433469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.433440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.596903 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.596868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.705177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.705097 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.808833 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.808797 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.907105 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:49.907064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:50.036429 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:50.036396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:50.611895 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:50.611858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:50.712225 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:50.712193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:53.020067 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:53.020036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:53.816872 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:53.816837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:35:55.242604 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:55.242566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:55.355154 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:55.355119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:56.513319 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:35:56.513254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.412583 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:01.412551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.555173 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:01.555139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.973283 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:01.973249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:02.072842 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:02.072805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:02.179865 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:02.179833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:03.571331 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:03.571286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:05.418286 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:05.418249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:36:09.571508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:09.571459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.679945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:09.679910 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.773830 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:09.773796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.875197 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:09.875111 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:10.337028 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:10.336988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:36.299808 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:36.299772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:42.233657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:42.233622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:42.305753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:42.305728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:42.388335 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:42.388300 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:58.361529 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:36:58.361443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:04.119886 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:04.119844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:04.226862 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:04.226813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:04.323026 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:04.322995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:14.859657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:14.859622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:15.718255 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:15.718223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:21.613550 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:21.613514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:21.716025 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:21.715994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:21.813159 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:21.813124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:21.915187 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:21.915104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:22.708092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:22.708058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:22.937755 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:22.937722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:24.540901 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:24.540869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:26.628167 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:26.628124 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:37:30.557479 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:30.557444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:30.657021 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:30.656987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:31.254189 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:31.254156 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:37.461776 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:37.461745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:37.561453 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:37.561424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:37.816797 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:37.816753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:38.141194 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:38.141104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:38.443375 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:38.443298 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:37:38.490043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:38.490013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:39.948998 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:39.948969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:40.058305 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:40.058277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:40.197955 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:40.197925 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:40.667186 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:40.667143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:40.739366 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:40.739333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:40.811814 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:40.811771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:45.479664 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:45.479613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:46.180524 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:46.180495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:46.283969 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:46.283939 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:46.635514 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:46.635482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:50.503229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:50.503197 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.021837 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:52.021806 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:37:52.870848 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:52.870816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.976962 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:52.976924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.631812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:53.631781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:54.222728 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:54.222699 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:54.323604 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:54.323574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:59.696910 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:59.696863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:59.775846 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:59.775807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:59.996146 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:37:59.996060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:00.209206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:00.209168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:00.339983 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:00.339949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:00.436067 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:00.436032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:00.551092 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:00.551058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:04.933084 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:04.933050 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:38:06.571824 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:06.571791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:06.683506 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:06.683471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:07.562338 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:07.562305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:07.663002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:07.662957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:13.030603 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:13.030569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:13.103141 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:13.103095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:20.775206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:20.775163 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:20.870273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:20.870239 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:20.940095 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:20.940061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:21.723900 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:21.723858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:21.822205 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:21.822167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:21.921310 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:21.921274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:23.342060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:23.342027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:23.828958 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:23.828924 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:38:29.388706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:29.388578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:29.458907 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:29.458872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:30.070563 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:30.070519 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.385135 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:33.385091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.485783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:33.485749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:35.966796 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:35.966759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:36.078075 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:36.078023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:36.167175 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:36.167139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:38.626136 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:38.626098 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:38:46.723306 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:46.723272 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:47.088667 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:38:47.088635 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:08.095472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:08.095437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:13.950954 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:13.950919 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:14.061927 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:14.061850 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:14.333534 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:14.333501 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:14.671043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:14.670950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:15.074388 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:15.074337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.192539 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.192505 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.287931 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.287901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.429538 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.429503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.578493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.578462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.895904 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.895820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:16.994832 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:16.994795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:18.824326 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:18.824285 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:39:21.740360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:21.740313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:22.074978 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:22.074937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:22.398881 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:22.398806 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:22.689476 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:22.689392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:24.271119 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:24.271088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:24.401852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:24.401818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:24.563839 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:24.563802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:24.665116 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:24.665084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:25.066956 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:25.066918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:25.170635 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:25.170601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.829173 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:29.829137 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:36.527509 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.527470 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:39:36.531164 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.531144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.535398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.535380 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da1123\"/\"default-dockercfg-mgp5n\"" Jul 15 21:39:36.536573 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.536556 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da1123\"/\"kube-root-ca.crt\"" Jul 15 21:39:36.536650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.536627 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da1123\"/\"test-model-da1123-kserve-self-signed-certs\"" Jul 15 21:39:36.536858 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.536659 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da1123\"/\"openshift-service-ca.crt\"" Jul 15 21:39:36.542006 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.541986 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:39:36.555919 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.555890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:36.616118 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.616279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.616279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.616279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jcl\" (UniqueName: \"kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.616417 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.616417 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.616309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jcl\" (UniqueName: \"kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717434 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717434 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717540 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.717596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.717567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.719421 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.719397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.719550 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.719533 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.725695 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.725650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jcl\" (UniqueName: \"kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl\") pod \"test-model-da1123-kserve-7779c55bf7-8zvpt\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.840369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.840321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:36.973494 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:36.973441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:39:36.976184 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:39:36.976157 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b59d37b_2a6c_4f46_9491_5a63d555447f.slice/crio-8b7f3e251a9c192c49fc7662338b3cc24131076d379a25df2ebe505737c1107d WatchSource:0}: Error finding container 8b7f3e251a9c192c49fc7662338b3cc24131076d379a25df2ebe505737c1107d: Status 404 returned error can't find the container with id 8b7f3e251a9c192c49fc7662338b3cc24131076d379a25df2ebe505737c1107d Jul 15 21:39:37.782562 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:37.782531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.881294 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:37.881260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.986532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:37.986496 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" event={"ID":"6b59d37b-2a6c-4f46-9491-5a63d555447f","Type":"ContainerStarted","Data":"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4"} Jul 15 21:39:37.986532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:37.986536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" event={"ID":"6b59d37b-2a6c-4f46-9491-5a63d555447f","Type":"ContainerStarted","Data":"8b7f3e251a9c192c49fc7662338b3cc24131076d379a25df2ebe505737c1107d"} Jul 15 21:39:38.006600 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:38.006552 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" podStartSLOduration=2.006536624 podStartE2EDuration="2.006536624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:39:38.005120413 +0000 UTC m=+1773.246266912" watchObservedRunningTime="2026-07-15 21:39:38.006536624 +0000 UTC m=+1773.247683111" Jul 15 21:39:39.215255 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:39.215218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:39:45.873373 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:45.873339 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:45.980151 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:45.980118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:46.841036 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:46.840998 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:46.841218 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:46.841059 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:46.853563 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:46.853536 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:47.037270 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:47.037246 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:39:56.141743 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:39:56.141712 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:40:05.322981 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:05.322946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:40:05.327993 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:05.327972 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:40:13.720124 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.720093 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:40:13.723327 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.723311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.726115 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.726097 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9625f6\"/\"kube-root-ca.crt\"" Jul 15 21:40:13.727348 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.727333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9625f6\"/\"default-dockercfg-nv7l4\"" Jul 15 21:40:13.727475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.727461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9625f6\"/\"openshift-service-ca.crt\"" Jul 15 21:40:13.727569 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.727507 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9625f6\"/\"test-model-9625f6-kserve-self-signed-certs\"" Jul 15 21:40:13.735083 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.735063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:40:13.748598 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.748571 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:13.852565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zbs\" (UniqueName: \"kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.852764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.852764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.852764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.852764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852722 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.852764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.852740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.953670 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.953851 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.953851 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zbs\" (UniqueName: \"kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.953851 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.953851 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.954025 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.953911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.954083 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.954062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.954130 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.954101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.954227 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.954211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.955923 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.955904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.956012 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.955994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:13.963145 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:13.963112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zbs\" (UniqueName: \"kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs\") pod \"test-model-9625f6-kserve-596dbdf997-vdl98\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:14.032150 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:14.032066 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:14.165060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:14.165033 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:40:14.167258 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:40:14.167231 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod484482d4_602a_4aab_8286_6a109e7cb1cc.slice/crio-f73f9d1f6c88d0e9b42a9a012eed181b1be360a832c0f103997916fc4763d0d1 WatchSource:0}: Error finding container f73f9d1f6c88d0e9b42a9a012eed181b1be360a832c0f103997916fc4763d0d1: Status 404 returned error can't find the container with id f73f9d1f6c88d0e9b42a9a012eed181b1be360a832c0f103997916fc4763d0d1 Jul 15 21:40:15.153369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:15.153334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" event={"ID":"484482d4-602a-4aab-8286-6a109e7cb1cc","Type":"ContainerStarted","Data":"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6"} Jul 15 21:40:15.153369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:15.153375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" event={"ID":"484482d4-602a-4aab-8286-6a109e7cb1cc","Type":"ContainerStarted","Data":"f73f9d1f6c88d0e9b42a9a012eed181b1be360a832c0f103997916fc4763d0d1"} Jul 15 21:40:15.177640 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:15.177585 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" podStartSLOduration=2.177570056 podStartE2EDuration="2.177570056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:40:15.176514671 +0000 UTC m=+1810.417661160" watchObservedRunningTime="2026-07-15 21:40:15.177570056 +0000 UTC m=+1810.418716577" Jul 15 21:40:17.831867 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:17.831829 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:40:20.824536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:20.824497 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:20.926659 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:20.926626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:24.033085 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:24.033043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:24.033484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:24.033326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:24.045597 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:24.045576 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:24.206060 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:24.206032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:40:33.021104 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:33.021068 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:40:43.789725 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:43.789693 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:40:43.809465 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:43.809438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:44.718596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:44.718561 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:40:44.726496 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:44.726455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:45.187632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:45.187595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:47.825830 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:47.825794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:40:50.269694 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:50.269600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:58.809417 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:58.809373 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" podUID="6b59d37b-2a6c-4f46-9491-5a63d555447f" containerName="main" containerID="cri-o://638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4" gracePeriod=45 Jul 15 21:40:59.063414 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.063355 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:40:59.160938 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.160899 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.160938 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.160941 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9jcl\" (UniqueName: \"kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.161202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.160962 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.161202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161014 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.161202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.161202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161121 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm\") pod \"6b59d37b-2a6c-4f46-9491-5a63d555447f\" (UID: \"6b59d37b-2a6c-4f46-9491-5a63d555447f\") " Jul 15 21:40:59.161397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:59.161397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home" (OuterVolumeSpecName: "home") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:59.161397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161307 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:59.161537 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161451 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.161537 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161464 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.161537 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.161478 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.163261 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.163235 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm" (OuterVolumeSpecName: "dshm") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:59.163261 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.163251 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:40:59.163418 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.163307 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl" (OuterVolumeSpecName: "kube-api-access-d9jcl") pod "6b59d37b-2a6c-4f46-9491-5a63d555447f" (UID: "6b59d37b-2a6c-4f46-9491-5a63d555447f"). InnerVolumeSpecName "kube-api-access-d9jcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:40:59.262870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.262838 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b59d37b-2a6c-4f46-9491-5a63d555447f-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.262870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.262862 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b59d37b-2a6c-4f46-9491-5a63d555447f-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.262870 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.262871 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jcl\" (UniqueName: \"kubernetes.io/projected/6b59d37b-2a6c-4f46-9491-5a63d555447f-kube-api-access-d9jcl\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:40:59.350749 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.350645 2574 generic.go:356] "Generic (PLEG): container finished" podID="6b59d37b-2a6c-4f46-9491-5a63d555447f" containerID="638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4" exitCode=0 Jul 15 21:40:59.350749 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.350706 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" Jul 15 21:40:59.350749 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.350729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" event={"ID":"6b59d37b-2a6c-4f46-9491-5a63d555447f","Type":"ContainerDied","Data":"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4"} Jul 15 21:40:59.351000 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.350776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt" event={"ID":"6b59d37b-2a6c-4f46-9491-5a63d555447f","Type":"ContainerDied","Data":"8b7f3e251a9c192c49fc7662338b3cc24131076d379a25df2ebe505737c1107d"} Jul 15 21:40:59.351000 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.350796 2574 scope.go:117] "RemoveContainer" containerID="638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4" Jul 15 21:40:59.364591 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.364571 2574 scope.go:117] "RemoveContainer" containerID="638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4" Jul 15 21:40:59.364888 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:40:59.364867 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4\": container with ID starting with 638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4 not found: ID does not exist" containerID="638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4" Jul 15 21:40:59.364987 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.364892 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4"} err="failed to get container status \"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4\": rpc error: code = NotFound desc = could not find container \"638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4\": container with ID starting with 638b039465eacbf5da043bed6af41d0b374f15fa65a9d94b6e576fa7893a47d4 not found: ID does not exist" Jul 15 21:40:59.375447 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.375416 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:40:59.378835 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.378816 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-da1123/test-model-da1123-kserve-7779c55bf7-8zvpt"] Jul 15 21:40:59.739252 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.739153 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" podUID="484482d4-602a-4aab-8286-6a109e7cb1cc" containerName="main" containerID="cri-o://bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6" gracePeriod=45 Jul 15 21:40:59.982715 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:40:59.982666 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:41:00.068908 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.068874 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069087 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.068920 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069087 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.068951 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069087 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8zbs\" (UniqueName: \"kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069087 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069069 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069303 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069113 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm\") pod \"484482d4-602a-4aab-8286-6a109e7cb1cc\" (UID: \"484482d4-602a-4aab-8286-6a109e7cb1cc\") " Jul 15 21:41:00.069303 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069251 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home" (OuterVolumeSpecName: "home") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:41:00.069303 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:41:00.069457 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069378 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:41:00.069457 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069431 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.069457 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.069449 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.071071 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.071048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:41:00.071174 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.071084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs" (OuterVolumeSpecName: "kube-api-access-p8zbs") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "kube-api-access-p8zbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:41:00.071174 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.071129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm" (OuterVolumeSpecName: "dshm") pod "484482d4-602a-4aab-8286-6a109e7cb1cc" (UID: "484482d4-602a-4aab-8286-6a109e7cb1cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:41:00.170206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.170170 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484482d4-602a-4aab-8286-6a109e7cb1cc-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.170206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.170195 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zbs\" (UniqueName: \"kubernetes.io/projected/484482d4-602a-4aab-8286-6a109e7cb1cc-kube-api-access-p8zbs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.170206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.170205 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.170206 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.170213 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484482d4-602a-4aab-8286-6a109e7cb1cc-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:41:00.360126 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.360040 2574 generic.go:356] "Generic (PLEG): container finished" podID="484482d4-602a-4aab-8286-6a109e7cb1cc" containerID="bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6" exitCode=0 Jul 15 21:41:00.360126 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.360081 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" Jul 15 21:41:00.360126 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.360089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" event={"ID":"484482d4-602a-4aab-8286-6a109e7cb1cc","Type":"ContainerDied","Data":"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6"} Jul 15 21:41:00.360126 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.360117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98" event={"ID":"484482d4-602a-4aab-8286-6a109e7cb1cc","Type":"ContainerDied","Data":"f73f9d1f6c88d0e9b42a9a012eed181b1be360a832c0f103997916fc4763d0d1"} Jul 15 21:41:00.360404 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.360133 2574 scope.go:117] "RemoveContainer" containerID="bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6" Jul 15 21:41:00.369109 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.369092 2574 scope.go:117] "RemoveContainer" containerID="bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6" Jul 15 21:41:00.369334 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:41:00.369319 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6\": container with ID starting with bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6 not found: ID does not exist" containerID="bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6" Jul 15 21:41:00.369380 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.369340 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6"} err="failed to get container status \"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6\": rpc error: code = NotFound desc = could not find container \"bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6\": container with ID starting with bd7f73a04154b7de05ec174880b8aed51b14ba16089e7a59dd7944a351aaece6 not found: ID does not exist" Jul 15 21:41:00.380981 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.380957 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:41:00.384725 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:00.384706 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9625f6/test-model-9625f6-kserve-596dbdf997-vdl98"] Jul 15 21:41:01.292213 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:01.292177 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484482d4-602a-4aab-8286-6a109e7cb1cc" path="/var/lib/kubelet/pods/484482d4-602a-4aab-8286-6a109e7cb1cc/volumes" Jul 15 21:41:01.292581 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:01.292545 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b59d37b-2a6c-4f46-9491-5a63d555447f" path="/var/lib/kubelet/pods/6b59d37b-2a6c-4f46-9491-5a63d555447f/volumes" Jul 15 21:41:04.406981 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:04.406948 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:04.682645 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:04.682567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:05.076091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:05.076059 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:05.381791 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:05.381763 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:08.021231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:08.021194 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:41:11.456391 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:11.456365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:11.526785 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:11.526755 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:11.730372 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:11.730296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:12.018401 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:12.018320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:12.332914 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:12.332877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:12.680051 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:12.679975 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:12.885007 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:12.884973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:14.923330 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:14.923277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:15.013536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:15.013503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:15.098429 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:15.098398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:15.206863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:15.206775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:15.621852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:15.621819 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:15.723161 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:15.723109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:20.438996 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:20.438958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:20.896403 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:20.896367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:21.350573 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:21.350541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:21.706506 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:21.706428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:22.953370 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:22.953337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.071688 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.071639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.161279 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.161246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.523929 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.523893 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:41:23.650577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.650547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.751289 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.751249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.855002 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:23.854969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:28.473126 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:28.473045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:36.950174 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:36.950139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:41.919983 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:41.919947 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:41:43.355920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:43.355892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:49.354977 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:49.354943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:49.430967 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:49.430914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:49.678665 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:49.678586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:49.930650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:49.930567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:50.385295 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:50.385259 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:53.814127 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:53.814096 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:41:56.349331 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:41:56.349294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:02.125945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:02.125907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:02.190229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:02.190185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:02.467919 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:02.467839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:02.634689 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:02.634633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.004082 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:03.004052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.351548 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:03.351514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.457398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:03.457365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:05.333797 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:05.333764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:05.538596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:05.538563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:05.627140 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:05.627053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:05.726990 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:05.726943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:06.114185 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:06.114125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:06.214733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:06.214656 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:08.919532 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:08.919500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:42:10.795361 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:10.795328 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:11.127005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:11.126918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:11.425876 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:11.425796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:11.555472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:11.555437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:13.288030 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:13.287994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:13.424313 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:13.424280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:13.639288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:13.639204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:13.928570 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:13.928499 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:14.264000 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:14.263923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:14.358057 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:14.358028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:18.846651 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:18.846621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:26.222619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:26.222585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:42:27.021914 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:27.021878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.556600 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:28.556555 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:29.263916 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.263884 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:42:29.264306 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264287 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484482d4-602a-4aab-8286-6a109e7cb1cc" containerName="main" Jul 15 21:42:29.264401 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264308 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="484482d4-602a-4aab-8286-6a109e7cb1cc" containerName="main" Jul 15 21:42:29.264401 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264322 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b59d37b-2a6c-4f46-9491-5a63d555447f" containerName="main" Jul 15 21:42:29.264401 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264332 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b59d37b-2a6c-4f46-9491-5a63d555447f" containerName="main" Jul 15 21:42:29.264552 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264413 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="484482d4-602a-4aab-8286-6a109e7cb1cc" containerName="main" Jul 15 21:42:29.264552 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.264426 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b59d37b-2a6c-4f46-9491-5a63d555447f" containerName="main" Jul 15 21:42:29.267501 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.267481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.270214 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.270196 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"default-dockercfg-h9kwp\"" Jul 15 21:42:29.270313 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.270241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"auth-test-model-543174-kserve-self-signed-certs\"" Jul 15 21:42:29.270472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.270450 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"openshift-service-ca.crt\"" Jul 15 21:42:29.270977 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.270959 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"kube-root-ca.crt\"" Jul 15 21:42:29.282593 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.282574 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:42:29.304360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.304338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:29.352661 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.352781 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.352781 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9nt\" (UniqueName: \"kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.352781 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.352886 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.352886 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.352876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9nt\" (UniqueName: \"kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454902 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.454951 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.454932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.456571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.456548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.456852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.456835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.475780 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.475757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9nt\" (UniqueName: \"kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt\") pod \"auth-test-model-543174-kserve-6dfcd4684c-cj7tx\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.577099 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.577071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:29.710620 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.710584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:42:29.751477 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:29.751448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" event={"ID":"efa00129-f5d2-459d-8366-fe862915a3c8","Type":"ContainerStarted","Data":"1b54fcdf1531d1aea7642c180d01d8bf28cb40a29635a4b26e7e1ea52a2b5758"} Jul 15 21:42:30.756934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:30.756900 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" event={"ID":"efa00129-f5d2-459d-8366-fe862915a3c8","Type":"ContainerStarted","Data":"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc"} Jul 15 21:42:30.778343 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:30.778286 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" podStartSLOduration=1.7782667220000001 podStartE2EDuration="1.778266722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:42:30.778221371 +0000 UTC m=+1946.019367871" watchObservedRunningTime="2026-07-15 21:42:30.778266722 +0000 UTC m=+1946.019413211" Jul 15 21:42:35.199459 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:35.199424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:35.302855 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:35.302794 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:38.826785 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:38.826753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:42:39.577603 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:39.577570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:39.577813 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:39.577614 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:39.589900 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:39.589880 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:39.806997 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:39.806965 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:42:50.123563 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.123533 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:42:50.126890 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.126873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.129749 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.129723 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"kube-root-ca.crt\"" Jul 15 21:42:50.129860 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.129795 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"openshift-service-ca.crt\"" Jul 15 21:42:50.130950 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.130932 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"default-dockercfg-l5wbj\"" Jul 15 21:42:50.131059 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.130952 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"auth-test-model-915a1c-kserve-self-signed-certs\"" Jul 15 21:42:50.139485 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.139461 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:42:50.152469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.152442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:50.229661 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtckq\" (UniqueName: \"kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.229786 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.229786 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.229856 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.229856 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.229948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.229896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.330761 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.330868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.330868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.330868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtckq\" (UniqueName: \"kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.331014 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.331014 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.330939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.331179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.331158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.331231 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.331202 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.331333 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.331317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.333111 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.333085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.333339 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.333323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.339573 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.339549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtckq\" (UniqueName: \"kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq\") pod \"auth-test-model-915a1c-kserve-556954b69f-2d69h\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.436444 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.436379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:42:50.568428 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.568402 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:42:50.570626 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:42:50.570602 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d1aba82_4da6_40b8_af67_778056385884.slice/crio-4a5c9424cd5aadd8f028c516dcbb4a27a33ceadb2e7ba16a3f9d15248957014e WatchSource:0}: Error finding container 4a5c9424cd5aadd8f028c516dcbb4a27a33ceadb2e7ba16a3f9d15248957014e: Status 404 returned error can't find the container with id 4a5c9424cd5aadd8f028c516dcbb4a27a33ceadb2e7ba16a3f9d15248957014e Jul 15 21:42:50.845977 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.845939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" event={"ID":"7d1aba82-4da6-40b8-af67-778056385884","Type":"ContainerStarted","Data":"ffc5d04f1b8693d0f670ed88e9623eecffc3e6526c83a13fe19ea1c71f198f94"} Jul 15 21:42:50.845977 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:50.845983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" event={"ID":"7d1aba82-4da6-40b8-af67-778056385884","Type":"ContainerStarted","Data":"4a5c9424cd5aadd8f028c516dcbb4a27a33ceadb2e7ba16a3f9d15248957014e"} Jul 15 21:42:53.924566 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:53.924516 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" podStartSLOduration=3.9244963999999998 podStartE2EDuration="3.9244964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:42:50.86421324 +0000 UTC m=+1966.105359738" watchObservedRunningTime="2026-07-15 21:42:53.9244964 +0000 UTC m=+1969.165642887" Jul 15 21:42:53.925132 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:42:53.925112 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:43:00.437414 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:00.437331 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:43:00.437414 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:00.437382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:43:00.450014 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:00.449993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:43:00.899430 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:00.899398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:43:13.122857 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:13.122818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:43:19.569979 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:19.569943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:31.308120 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:31.308081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:31.368338 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:31.368305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:31.412665 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:31.412623 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:43:34.229920 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:34.229886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:34.477949 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:34.477879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:34.569873 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:34.569822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:47.919318 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:43:47.919288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:44:07.816830 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:44:07.816796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:44:24.508362 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:44:24.508327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:44:27.621652 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:44:27.621570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:44:48.420649 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:44:48.420611 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:45:00.167425 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.167388 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9"] Jul 15 21:45:00.170480 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.170462 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8"] Jul 15 21:45:00.170621 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.170597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" Jul 15 21:45:00.173439 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.173421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" Jul 15 21:45:00.175339 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.175321 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6nvd\"" Jul 15 21:45:00.180609 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.180590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" Jul 15 21:45:00.186568 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.186552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" Jul 15 21:45:00.213750 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.213711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9"] Jul 15 21:45:00.218364 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.218279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8"] Jul 15 21:45:00.359311 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.359288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8"] Jul 15 21:45:00.361801 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:45:00.361775 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bfa61b_6cf2_4c67_b5ca_69645e7af648.slice/crio-69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846 WatchSource:0}: Error finding container 69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846: Status 404 returned error can't find the container with id 69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846 Jul 15 21:45:00.363716 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.363697 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:45:00.430280 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.430191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" event={"ID":"a8bfa61b-6cf2-4c67-b5ca-69645e7af648","Type":"ContainerStarted","Data":"69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846"} Jul 15 21:45:00.545849 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:00.545823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9"] Jul 15 21:45:00.548214 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:45:00.548188 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37009bb_abf1_4c26_8e48_89813a87fb37.slice/crio-a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221 WatchSource:0}: Error finding container a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221: Status 404 returned error can't find the container with id a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221 Jul 15 21:45:01.438640 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:01.438589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" event={"ID":"b37009bb-abf1-4c26-8e48-89813a87fb37","Type":"ContainerStarted","Data":"a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221"} Jul 15 21:45:03.449275 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:03.449233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" event={"ID":"a8bfa61b-6cf2-4c67-b5ca-69645e7af648","Type":"ContainerStarted","Data":"fdea9882534fb390dd5aed69d33b8954a76ebbac8c4843506873baf37c2dd193"} Jul 15 21:45:03.450643 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:03.450618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" event={"ID":"b37009bb-abf1-4c26-8e48-89813a87fb37","Type":"ContainerStarted","Data":"005a444491939788d7dd19500db447ac9bb16bbc092a5ce3ba199049420a3f7a"} Jul 15 21:45:03.466737 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:03.466686 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" podStartSLOduration=1.201414164 podStartE2EDuration="3.466656385s" totalImagesPullingTime="2.265242221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:45:03.465808918 +0000 UTC m=+2098.706955427" watchObservedRunningTime="2026-07-15 21:45:03.466656385 +0000 UTC m=+2098.707802921" Jul 15 21:45:03.483753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:03.483707 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" podStartSLOduration=1.401361432 podStartE2EDuration="3.483690179s" totalImagesPullingTime="2.082328747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:45:03.4825725 +0000 UTC m=+2098.723718987" watchObservedRunningTime="2026-07-15 21:45:03.483690179 +0000 UTC m=+2098.724836658" Jul 15 21:45:04.457096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:04.457063 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8bfa61b-6cf2-4c67-b5ca-69645e7af648" containerID="fdea9882534fb390dd5aed69d33b8954a76ebbac8c4843506873baf37c2dd193" exitCode=0 Jul 15 21:45:04.457549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:04.457138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" event={"ID":"a8bfa61b-6cf2-4c67-b5ca-69645e7af648","Type":"ContainerDied","Data":"fdea9882534fb390dd5aed69d33b8954a76ebbac8c4843506873baf37c2dd193"} Jul 15 21:45:04.458628 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:04.458606 2574 generic.go:356] "Generic (PLEG): container finished" podID="b37009bb-abf1-4c26-8e48-89813a87fb37" containerID="005a444491939788d7dd19500db447ac9bb16bbc092a5ce3ba199049420a3f7a" exitCode=0 Jul 15 21:45:04.458744 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:04.458690 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" event={"ID":"b37009bb-abf1-4c26-8e48-89813a87fb37","Type":"ContainerDied","Data":"005a444491939788d7dd19500db447ac9bb16bbc092a5ce3ba199049420a3f7a"} Jul 15 21:45:05.351497 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:05.351468 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:45:05.356706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:05.356665 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pjmhv_e8622b0a-797d-4b91-917a-ea4f58d59733/ovn-acl-logging/0.log" Jul 15 21:45:05.607469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:05.607402 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" Jul 15 21:45:05.609963 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:05.609946 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" Jul 15 21:45:05.823818 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:05.823782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:45:06.470596 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.470569 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" Jul 15 21:45:06.470816 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.470567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8" event={"ID":"a8bfa61b-6cf2-4c67-b5ca-69645e7af648","Type":"ContainerDied","Data":"69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846"} Jul 15 21:45:06.470816 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.470707 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fb0109c6ed136757587a7c89858a7e43a038f134114db286eff1bbf171b846" Jul 15 21:45:06.471863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.471842 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" Jul 15 21:45:06.471970 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.471878 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735865-smkj9" event={"ID":"b37009bb-abf1-4c26-8e48-89813a87fb37","Type":"ContainerDied","Data":"a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221"} Jul 15 21:45:06.471970 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:06.471918 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7757549392b4a551729ba36128edfb533c459a9d0206b91da64282a50e33221" Jul 15 21:45:09.097177 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:09.097137 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:45:09.124377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:09.124349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:10.342549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:10.342516 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:45:10.380733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:10.380702 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:17.738170 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738134 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:45:17.738531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bfa61b-6cf2-4c67-b5ca-69645e7af648" containerName="cleanup" Jul 15 21:45:17.738531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738498 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bfa61b-6cf2-4c67-b5ca-69645e7af648" containerName="cleanup" Jul 15 21:45:17.738531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738517 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37009bb-abf1-4c26-8e48-89813a87fb37" containerName="cleanup" Jul 15 21:45:17.738531 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738523 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37009bb-abf1-4c26-8e48-89813a87fb37" containerName="cleanup" Jul 15 21:45:17.738697 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738584 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37009bb-abf1-4c26-8e48-89813a87fb37" containerName="cleanup" Jul 15 21:45:17.738697 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.738593 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bfa61b-6cf2-4c67-b5ca-69645e7af648" containerName="cleanup" Jul 15 21:45:17.741582 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.741564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.744423 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.744399 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"sub-test-model-543174-kserve-self-signed-certs\"" Jul 15 21:45:17.751171 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.751147 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:45:17.767288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.767256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:17.831429 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.831429 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.831644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p687m\" (UniqueName: \"kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.831644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.831644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.831644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.831632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.932410 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.932410 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p687m\" (UniqueName: \"kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.932841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.934761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.935188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:17.951180 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:17.943036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p687m\" (UniqueName: \"kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m\") pod \"sub-test-model-543174-kserve-68fdd4d459-nntnv\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:18.051274 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:18.051246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:18.185874 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:18.185746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:45:18.188615 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:45:18.188590 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3b88b97_b119_4b5b_a865_f5663052ff1e.slice/crio-82df69edff7c4e92aa9b532999689973145c7bb1198d1a78cff6b057d2198109 WatchSource:0}: Error finding container 82df69edff7c4e92aa9b532999689973145c7bb1198d1a78cff6b057d2198109: Status 404 returned error can't find the container with id 82df69edff7c4e92aa9b532999689973145c7bb1198d1a78cff6b057d2198109 Jul 15 21:45:18.530777 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:18.530665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" event={"ID":"a3b88b97-b119-4b5b-a865-f5663052ff1e","Type":"ContainerStarted","Data":"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d"} Jul 15 21:45:18.530777 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:18.530730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" event={"ID":"a3b88b97-b119-4b5b-a865-f5663052ff1e","Type":"ContainerStarted","Data":"82df69edff7c4e92aa9b532999689973145c7bb1198d1a78cff6b057d2198109"} Jul 15 21:45:18.556192 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:18.556140 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" podStartSLOduration=1.55612314 podStartE2EDuration="1.55612314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:45:18.554573457 +0000 UTC m=+2113.795719944" watchObservedRunningTime="2026-07-15 21:45:18.55612314 +0000 UTC m=+2113.797269627" Jul 15 21:45:23.818543 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:23.818502 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:45:24.117620 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.117520 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" podUID="efa00129-f5d2-459d-8366-fe862915a3c8" containerName="main" containerID="cri-o://8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc" gracePeriod=45 Jul 15 21:45:24.380179 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.380119 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:45:24.394609 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394644 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394730 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394698 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394812 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394865 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394852 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394903 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9nt\" (UniqueName: \"kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt\") pod \"efa00129-f5d2-459d-8366-fe862915a3c8\" (UID: \"efa00129-f5d2-459d-8366-fe862915a3c8\") " Jul 15 21:45:24.394952 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394890 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:24.394952 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394906 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home" (OuterVolumeSpecName: "home") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:24.394952 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.394912 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:24.395169 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.395143 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.395169 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.395166 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.395315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.395179 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.397009 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.396975 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:45:24.397130 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.397106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm" (OuterVolumeSpecName: "dshm") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:24.397175 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.397115 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt" (OuterVolumeSpecName: "kube-api-access-2f9nt") pod "efa00129-f5d2-459d-8366-fe862915a3c8" (UID: "efa00129-f5d2-459d-8366-fe862915a3c8"). InnerVolumeSpecName "kube-api-access-2f9nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:45:24.495878 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.495846 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa00129-f5d2-459d-8366-fe862915a3c8-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.495878 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.495874 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9nt\" (UniqueName: \"kubernetes.io/projected/efa00129-f5d2-459d-8366-fe862915a3c8-kube-api-access-2f9nt\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.495878 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.495883 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa00129-f5d2-459d-8366-fe862915a3c8-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:24.561242 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.561200 2574 generic.go:356] "Generic (PLEG): container finished" podID="efa00129-f5d2-459d-8366-fe862915a3c8" containerID="8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc" exitCode=0 Jul 15 21:45:24.561431 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.561250 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" Jul 15 21:45:24.561431 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.561278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" event={"ID":"efa00129-f5d2-459d-8366-fe862915a3c8","Type":"ContainerDied","Data":"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc"} Jul 15 21:45:24.561431 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.561328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx" event={"ID":"efa00129-f5d2-459d-8366-fe862915a3c8","Type":"ContainerDied","Data":"1b54fcdf1531d1aea7642c180d01d8bf28cb40a29635a4b26e7e1ea52a2b5758"} Jul 15 21:45:24.561431 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.561345 2574 scope.go:117] "RemoveContainer" containerID="8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc" Jul 15 21:45:24.578237 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.578208 2574 scope.go:117] "RemoveContainer" containerID="8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc" Jul 15 21:45:24.578540 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:45:24.578517 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc\": container with ID starting with 8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc not found: ID does not exist" containerID="8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc" Jul 15 21:45:24.578702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.578548 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc"} err="failed to get container status \"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc\": rpc error: code = NotFound desc = could not find container \"8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc\": container with ID starting with 8124ced9bcf7fd8d581ca2980a4ef88218ae17bd9c87b02afd108e9fbe9ad2dc not found: ID does not exist" Jul 15 21:45:24.589110 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.589081 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:45:24.596219 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:24.596186 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-543174/auth-test-model-543174-kserve-6dfcd4684c-cj7tx"] Jul 15 21:45:25.293796 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.293765 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa00129-f5d2-459d-8366-fe862915a3c8" path="/var/lib/kubelet/pods/efa00129-f5d2-459d-8366-fe862915a3c8/volumes" Jul 15 21:45:25.362404 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.362362 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" podUID="7d1aba82-4da6-40b8-af67-778056385884" containerName="main" containerID="cri-o://ffc5d04f1b8693d0f670ed88e9623eecffc3e6526c83a13fe19ea1c71f198f94" gracePeriod=45 Jul 15 21:45:25.567146 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.567063 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d1aba82-4da6-40b8-af67-778056385884" containerID="ffc5d04f1b8693d0f670ed88e9623eecffc3e6526c83a13fe19ea1c71f198f94" exitCode=0 Jul 15 21:45:25.567284 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.567127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" event={"ID":"7d1aba82-4da6-40b8-af67-778056385884","Type":"ContainerDied","Data":"ffc5d04f1b8693d0f670ed88e9623eecffc3e6526c83a13fe19ea1c71f198f94"} Jul 15 21:45:25.609732 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.609706 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:45:25.706757 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706726 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.706945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.706945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706836 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtckq\" (UniqueName: \"kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.706945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706883 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.706945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.706945 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.706943 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs\") pod \"7d1aba82-4da6-40b8-af67-778056385884\" (UID: \"7d1aba82-4da6-40b8-af67-778056385884\") " Jul 15 21:45:25.707229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:25.707229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:25.707229 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707149 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home" (OuterVolumeSpecName: "home") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:25.707366 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707241 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:25.707366 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707255 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:25.707366 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.707269 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:25.709251 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.709220 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm" (OuterVolumeSpecName: "dshm") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:25.709357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.709275 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:45:25.709357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.709308 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq" (OuterVolumeSpecName: "kube-api-access-xtckq") pod "7d1aba82-4da6-40b8-af67-778056385884" (UID: "7d1aba82-4da6-40b8-af67-778056385884"). InnerVolumeSpecName "kube-api-access-xtckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:45:25.808533 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.808496 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtckq\" (UniqueName: \"kubernetes.io/projected/7d1aba82-4da6-40b8-af67-778056385884-kube-api-access-xtckq\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:25.808533 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.808526 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1aba82-4da6-40b8-af67-778056385884-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:25.808533 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:25.808537 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1aba82-4da6-40b8-af67-778056385884-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:45:26.574153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:26.574124 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" Jul 15 21:45:26.574153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:26.574128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h" event={"ID":"7d1aba82-4da6-40b8-af67-778056385884","Type":"ContainerDied","Data":"4a5c9424cd5aadd8f028c516dcbb4a27a33ceadb2e7ba16a3f9d15248957014e"} Jul 15 21:45:26.574639 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:26.574181 2574 scope.go:117] "RemoveContainer" containerID="ffc5d04f1b8693d0f670ed88e9623eecffc3e6526c83a13fe19ea1c71f198f94" Jul 15 21:45:26.595632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:26.595595 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:45:26.601762 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:26.601736 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/auth-test-model-915a1c-kserve-556954b69f-2d69h"] Jul 15 21:45:27.293304 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:27.293271 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1aba82-4da6-40b8-af67-778056385884" path="/var/lib/kubelet/pods/7d1aba82-4da6-40b8-af67-778056385884/volumes" Jul 15 21:45:28.052241 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:28.052193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:28.052241 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:28.052252 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:28.064855 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:28.064829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:28.596005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:28.595975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:45:32.553805 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:32.553769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:41.920086 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920040 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:45:41.920629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920612 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa00129-f5d2-459d-8366-fe862915a3c8" containerName="main" Jul 15 21:45:41.920692 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa00129-f5d2-459d-8366-fe862915a3c8" containerName="main" Jul 15 21:45:41.920692 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920652 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1aba82-4da6-40b8-af67-778056385884" containerName="main" Jul 15 21:45:41.920692 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920661 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1aba82-4da6-40b8-af67-778056385884" containerName="main" Jul 15 21:45:41.920788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920754 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1aba82-4da6-40b8-af67-778056385884" containerName="main" Jul 15 21:45:41.920788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.920774 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa00129-f5d2-459d-8366-fe862915a3c8" containerName="main" Jul 15 21:45:41.924208 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.924191 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:41.928471 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.928446 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"default-dockercfg-l5wbj\"" Jul 15 21:45:41.929106 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.928470 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"sub-test-model-915a1c-kserve-self-signed-certs\"" Jul 15 21:45:41.929216 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.928548 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"openshift-service-ca.crt\"" Jul 15 21:45:41.929670 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.929645 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"kube-root-ca.crt\"" Jul 15 21:45:41.931578 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.931397 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:45:41.950939 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:41.950907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:42.053841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.054024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.054024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.054024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.054024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.054024 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.053939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgbd\" (UniqueName: \"kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155181 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgbd\" (UniqueName: \"kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.155702 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.156018 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.155769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.157577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.157556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.157762 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.157745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.164267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.164243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgbd\" (UniqueName: \"kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd\") pod \"sub-test-model-915a1c-kserve-b99f66bcf-bphg5\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.235018 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.234911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:42.373987 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.373953 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:45:42.377089 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:45:42.377055 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a6fe86f_257b_44b3_9d68_b201ffd41f8e.slice/crio-041c6480303e4f658362cb5f7e9fd4e89ec4ba2c722d53bc7d9d6e2441fc08a7 WatchSource:0}: Error finding container 041c6480303e4f658362cb5f7e9fd4e89ec4ba2c722d53bc7d9d6e2441fc08a7: Status 404 returned error can't find the container with id 041c6480303e4f658362cb5f7e9fd4e89ec4ba2c722d53bc7d9d6e2441fc08a7 Jul 15 21:45:42.651172 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.651132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" event={"ID":"8a6fe86f-257b-44b3-9d68-b201ffd41f8e","Type":"ContainerStarted","Data":"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747"} Jul 15 21:45:42.651172 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.651175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" event={"ID":"8a6fe86f-257b-44b3-9d68-b201ffd41f8e","Type":"ContainerStarted","Data":"041c6480303e4f658362cb5f7e9fd4e89ec4ba2c722d53bc7d9d6e2441fc08a7"} Jul 15 21:45:42.670475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:42.670423 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" podStartSLOduration=1.670405634 podStartE2EDuration="1.670405634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:45:42.670111521 +0000 UTC m=+2137.911258010" watchObservedRunningTime="2026-07-15 21:45:42.670405634 +0000 UTC m=+2137.911552123" Jul 15 21:45:47.031293 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:47.031253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:45:52.235268 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:52.235228 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:52.235746 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:52.235309 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:52.248111 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:52.248083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:45:52.714594 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:45:52.714562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:46:09.914546 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:09.914506 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:46:30.913897 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:30.913860 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:46:37.028577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:37.028530 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:46:37.049642 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:37.049612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:38.025951 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:38.025911 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:46:38.051656 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:38.051629 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:41.729802 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.729770 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:46:41.734752 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.733724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.737189 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.737160 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-543174\"/\"rate-test-model-543174-kserve-self-signed-certs\"" Jul 15 21:46:41.744044 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.744020 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:46:41.754934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.754904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:41.886846 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.886798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.886846 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.886852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.887096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.886926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.887096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.886974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phphp\" (UniqueName: \"kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.887096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.886995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.887096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.887043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988481 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988481 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phphp\" (UniqueName: \"kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988703 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.988948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.988868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.989041 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.989019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.989103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.989077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.990767 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.990746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.991023 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.990998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:41.997530 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:41.997509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phphp\" (UniqueName: \"kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp\") pod \"rate-test-model-543174-kserve-5f6854499f-slxgv\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:42.044354 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:42.044305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:42.178264 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:42.178239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:46:42.180667 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:46:42.180641 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0c9e436_bb85_480c_b0bc_897d75914f0a.slice/crio-9da1d1980446613cb9e0be19fbb3a329e228ce5fd541bbfaa456c15944187fea WatchSource:0}: Error finding container 9da1d1980446613cb9e0be19fbb3a329e228ce5fd541bbfaa456c15944187fea: Status 404 returned error can't find the container with id 9da1d1980446613cb9e0be19fbb3a329e228ce5fd541bbfaa456c15944187fea Jul 15 21:46:42.933056 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:42.933017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" event={"ID":"e0c9e436-bb85-480c-b0bc-897d75914f0a","Type":"ContainerStarted","Data":"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893"} Jul 15 21:46:42.933439 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:42.933067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" event={"ID":"e0c9e436-bb85-480c-b0bc-897d75914f0a","Type":"ContainerStarted","Data":"9da1d1980446613cb9e0be19fbb3a329e228ce5fd541bbfaa456c15944187fea"} Jul 15 21:46:42.953597 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:42.953549 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" podStartSLOduration=1.953533346 podStartE2EDuration="1.953533346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:46:42.950184031 +0000 UTC m=+2198.191330518" watchObservedRunningTime="2026-07-15 21:46:42.953533346 +0000 UTC m=+2198.194679833" Jul 15 21:46:51.833535 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:51.833495 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:46:52.045349 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.045307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:52.045523 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.045359 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:52.049118 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.049080 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" podUID="a3b88b97-b119-4b5b-a865-f5663052ff1e" containerName="main" containerID="cri-o://324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d" gracePeriod=45 Jul 15 21:46:52.058889 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.058862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:52.303840 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.303814 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:46:52.487560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.487560 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487551 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.487834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487607 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.487834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p687m\" (UniqueName: \"kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.487834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.487834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487772 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir\") pod \"a3b88b97-b119-4b5b-a865-f5663052ff1e\" (UID: \"a3b88b97-b119-4b5b-a865-f5663052ff1e\") " Jul 15 21:46:52.488035 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487859 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:52.488035 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.487869 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home" (OuterVolumeSpecName: "home") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:52.488137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.488039 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:52.488137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.488097 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.488137 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.488115 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.489778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.489756 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:46:52.489876 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.489856 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m" (OuterVolumeSpecName: "kube-api-access-p687m") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "kube-api-access-p687m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:46:52.489876 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.489865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm" (OuterVolumeSpecName: "dshm") pod "a3b88b97-b119-4b5b-a865-f5663052ff1e" (UID: "a3b88b97-b119-4b5b-a865-f5663052ff1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:52.589392 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.589359 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.589392 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.589388 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p687m\" (UniqueName: \"kubernetes.io/projected/a3b88b97-b119-4b5b-a865-f5663052ff1e-kube-api-access-p687m\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.589392 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.589398 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b88b97-b119-4b5b-a865-f5663052ff1e-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.589392 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.589407 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b88b97-b119-4b5b-a865-f5663052ff1e-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:52.979561 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.979523 2574 generic.go:356] "Generic (PLEG): container finished" podID="a3b88b97-b119-4b5b-a865-f5663052ff1e" containerID="324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d" exitCode=0 Jul 15 21:46:52.980026 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.979585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" Jul 15 21:46:52.980026 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.979612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" event={"ID":"a3b88b97-b119-4b5b-a865-f5663052ff1e","Type":"ContainerDied","Data":"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d"} Jul 15 21:46:52.980026 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.979656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv" event={"ID":"a3b88b97-b119-4b5b-a865-f5663052ff1e","Type":"ContainerDied","Data":"82df69edff7c4e92aa9b532999689973145c7bb1198d1a78cff6b057d2198109"} Jul 15 21:46:52.980026 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.979692 2574 scope.go:117] "RemoveContainer" containerID="324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d" Jul 15 21:46:52.989648 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.989628 2574 scope.go:117] "RemoveContainer" containerID="324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d" Jul 15 21:46:52.989949 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:46:52.989931 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d\": container with ID starting with 324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d not found: ID does not exist" containerID="324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d" Jul 15 21:46:52.990005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.989958 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d"} err="failed to get container status \"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d\": rpc error: code = NotFound desc = could not find container \"324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d\": container with ID starting with 324eff23a8112d3a1c7c85b31e37ef1fd2e89eb53ba412d38868e93eee3e805d not found: ID does not exist" Jul 15 21:46:52.993371 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:52.993352 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:46:53.001908 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.001883 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:46:53.005513 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.005491 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-543174/sub-test-model-543174-kserve-68fdd4d459-nntnv"] Jul 15 21:46:53.046215 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.046171 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" podUID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" containerName="main" containerID="cri-o://3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747" gracePeriod=45 Jul 15 21:46:53.293946 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.293915 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b88b97-b119-4b5b-a865-f5663052ff1e" path="/var/lib/kubelet/pods/a3b88b97-b119-4b5b-a865-f5663052ff1e/volumes" Jul 15 21:46:53.312183 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.312154 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:46:53.499440 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tgbd\" (UniqueName: \"kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499462 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499495 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499521 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499644 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499581 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache\") pod \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\" (UID: \"8a6fe86f-257b-44b3-9d68-b201ffd41f8e\") " Jul 15 21:46:53.499947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499802 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:53.499947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499834 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home" (OuterVolumeSpecName: "home") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:53.499947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.499916 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:53.501642 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.501615 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm" (OuterVolumeSpecName: "dshm") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:46:53.501778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.501695 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd" (OuterVolumeSpecName: "kube-api-access-5tgbd") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "kube-api-access-5tgbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:46:53.501778 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.501714 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a6fe86f-257b-44b3-9d68-b201ffd41f8e" (UID: "8a6fe86f-257b-44b3-9d68-b201ffd41f8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600571 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600603 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600619 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgbd\" (UniqueName: \"kubernetes.io/projected/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-kube-api-access-5tgbd\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600630 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600642 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.600700 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.600651 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6fe86f-257b-44b3-9d68-b201ffd41f8e-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:46:53.986783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.986666 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" containerID="3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747" exitCode=0 Jul 15 21:46:53.986783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.986731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" Jul 15 21:46:53.987315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.986775 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" event={"ID":"8a6fe86f-257b-44b3-9d68-b201ffd41f8e","Type":"ContainerDied","Data":"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747"} Jul 15 21:46:53.987315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.986823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5" event={"ID":"8a6fe86f-257b-44b3-9d68-b201ffd41f8e","Type":"ContainerDied","Data":"041c6480303e4f658362cb5f7e9fd4e89ec4ba2c722d53bc7d9d6e2441fc08a7"} Jul 15 21:46:53.987315 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.986841 2574 scope.go:117] "RemoveContainer" containerID="3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747" Jul 15 21:46:53.995599 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.995581 2574 scope.go:117] "RemoveContainer" containerID="3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747" Jul 15 21:46:53.995887 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:46:53.995868 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747\": container with ID starting with 3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747 not found: ID does not exist" containerID="3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747" Jul 15 21:46:53.995957 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:53.995909 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747"} err="failed to get container status \"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747\": rpc error: code = NotFound desc = could not find container \"3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747\": container with ID starting with 3a29736c6c0c8104fa73f80be08653c9626d09b28e34ab1c4f207a3092647747 not found: ID does not exist" Jul 15 21:46:54.009153 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:54.009129 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:46:54.012230 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:54.012210 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/sub-test-model-915a1c-kserve-b99f66bcf-bphg5"] Jul 15 21:46:55.293412 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:55.293379 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" path="/var/lib/kubelet/pods/8a6fe86f-257b-44b3-9d68-b201ffd41f8e/volumes" Jul 15 21:46:59.526508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526472 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526804 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" containerName="main" Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526815 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" containerName="main" Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526828 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b88b97-b119-4b5b-a865-f5663052ff1e" containerName="main" Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526833 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b88b97-b119-4b5b-a865-f5663052ff1e" containerName="main" Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526895 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b88b97-b119-4b5b-a865-f5663052ff1e" containerName="main" Jul 15 21:46:59.526948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.526903 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6fe86f-257b-44b3-9d68-b201ffd41f8e" containerName="main" Jul 15 21:46:59.530574 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.530558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.534866 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.534836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"default-dockercfg-l5wbj\"" Jul 15 21:46:59.535005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.534863 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"openshift-service-ca.crt\"" Jul 15 21:46:59.535005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.534893 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"rate-test-model-915a1c-kserve-self-signed-certs\"" Jul 15 21:46:59.535005 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.534897 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-915a1c\"/\"kube-root-ca.crt\"" Jul 15 21:46:59.539587 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.539563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550114 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhgz\" (UniqueName: \"kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.550713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.550305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.620091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.620057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:59.651072 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651269 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651269 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651269 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhgz\" (UniqueName: \"kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651269 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651502 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651502 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.651733 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.651720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.653402 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.653384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.653653 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.653638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.659687 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.659660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhgz\" (UniqueName: \"kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz\") pod \"rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.840484 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.840449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:46:59.978573 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:46:59.978546 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:46:59.980890 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:46:59.980850 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8750b68c_670a_42f2_b189_51ddbb35b22b.slice/crio-acae3e46f5f7aab596d350371d23ed84487714841da41ffa2dc754c0e2685caa WatchSource:0}: Error finding container acae3e46f5f7aab596d350371d23ed84487714841da41ffa2dc754c0e2685caa: Status 404 returned error can't find the container with id acae3e46f5f7aab596d350371d23ed84487714841da41ffa2dc754c0e2685caa Jul 15 21:47:00.019267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:00.019093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" event={"ID":"8750b68c-670a-42f2-b189-51ddbb35b22b","Type":"ContainerStarted","Data":"acae3e46f5f7aab596d350371d23ed84487714841da41ffa2dc754c0e2685caa"} Jul 15 21:47:01.025125 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:01.025091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" event={"ID":"8750b68c-670a-42f2-b189-51ddbb35b22b","Type":"ContainerStarted","Data":"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d"} Jul 15 21:47:01.045638 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:01.045585 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" podStartSLOduration=2.045569423 podStartE2EDuration="2.045569423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:47:01.043818367 +0000 UTC m=+2216.284964856" watchObservedRunningTime="2026-07-15 21:47:01.045569423 +0000 UTC m=+2216.286715909" Jul 15 21:47:06.824437 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:06.824396 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:47:09.841015 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:09.840972 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:47:09.841500 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:09.841031 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:47:09.853836 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:09.853800 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:47:10.078065 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:10.078034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:47:23.121780 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:23.121732 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:47:27.977728 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:27.977618 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:28.075383 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:28.075348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:35.991282 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:35.991248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:37.621969 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:37.621926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:41.225670 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:41.225631 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:47:48.360709 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:48.360664 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:47:48.414895 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:48.414857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:49.339808 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:49.339772 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:47:49.365017 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:49.364969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:50.335991 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.335957 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664773636656"] Jul 15 21:47:50.339246 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.339229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:50.341896 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.341877 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fqkxv\"" Jul 15 21:47:50.346500 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.346168 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664773636656"] Jul 15 21:47:50.415612 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.415576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982bv\" (UniqueName: \"kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv\") pod \"test-curl-7545-139664773636656\" (UID: \"6ec4c847-50d3-4888-90e1-29f30b6bbed6\") " pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:50.516630 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.516592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982bv\" (UniqueName: \"kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv\") pod \"test-curl-7545-139664773636656\" (UID: \"6ec4c847-50d3-4888-90e1-29f30b6bbed6\") " pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:50.526569 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.526527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-982bv\" (UniqueName: \"kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv\") pod \"test-curl-7545-139664773636656\" (UID: \"6ec4c847-50d3-4888-90e1-29f30b6bbed6\") " pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:50.648929 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.648834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:50.782422 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:50.782398 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664773636656"] Jul 15 21:47:50.784723 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:47:50.784695 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ec4c847_50d3_4888_90e1_29f30b6bbed6.slice/crio-f456bed359753a57713f160c087c3d81634fb16904aca612c631b795b2bffe06 WatchSource:0}: Error finding container f456bed359753a57713f160c087c3d81634fb16904aca612c631b795b2bffe06: Status 404 returned error can't find the container with id f456bed359753a57713f160c087c3d81634fb16904aca612c631b795b2bffe06 Jul 15 21:47:51.260632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:51.260591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664773636656" event={"ID":"6ec4c847-50d3-4888-90e1-29f30b6bbed6","Type":"ContainerStarted","Data":"f456bed359753a57713f160c087c3d81634fb16904aca612c631b795b2bffe06"} Jul 15 21:47:52.266871 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:52.266826 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" containerID="7ec815321c7abf6c4c97a1ff5fa56926475c47bf6ee38ed25b2faf3927fa3b01" exitCode=0 Jul 15 21:47:52.267266 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:52.266884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664773636656" event={"ID":"6ec4c847-50d3-4888-90e1-29f30b6bbed6","Type":"ContainerDied","Data":"7ec815321c7abf6c4c97a1ff5fa56926475c47bf6ee38ed25b2faf3927fa3b01"} Jul 15 21:47:53.407868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.407839 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:53.429016 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.428983 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664773636656_6ec4c847-50d3-4888-90e1-29f30b6bbed6/test-curl-7545-139664773636656/0.log" Jul 15 21:47:53.444360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.444334 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-982bv\" (UniqueName: \"kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv\") pod \"6ec4c847-50d3-4888-90e1-29f30b6bbed6\" (UID: \"6ec4c847-50d3-4888-90e1-29f30b6bbed6\") " Jul 15 21:47:53.446469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.446443 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv" (OuterVolumeSpecName: "kube-api-access-982bv") pod "6ec4c847-50d3-4888-90e1-29f30b6bbed6" (UID: "6ec4c847-50d3-4888-90e1-29f30b6bbed6"). InnerVolumeSpecName "kube-api-access-982bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:47:53.451134 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.451108 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664773636656"] Jul 15 21:47:53.453708 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.453666 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664773636656"] Jul 15 21:47:53.545783 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.545738 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982bv\" (UniqueName: \"kubernetes.io/projected/6ec4c847-50d3-4888-90e1-29f30b6bbed6-kube-api-access-982bv\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:47:53.573356 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.573313 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664773637296"] Jul 15 21:47:53.573750 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.573731 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" containerName="test-curl-7545-139664773636656" Jul 15 21:47:53.573852 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.573752 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" containerName="test-curl-7545-139664773636656" Jul 15 21:47:53.573907 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.573853 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" containerName="test-curl-7545-139664773636656" Jul 15 21:47:53.576826 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.576802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:53.584295 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.584271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664773637296"] Jul 15 21:47:53.646776 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.646730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77lx\" (UniqueName: \"kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx\") pod \"test-curl-7545-139664773637296\" (UID: \"3f77417d-34f0-438d-aba7-36f84093b3c3\") " pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:53.748133 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.748096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77lx\" (UniqueName: \"kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx\") pod \"test-curl-7545-139664773637296\" (UID: \"3f77417d-34f0-438d-aba7-36f84093b3c3\") " pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:53.757241 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.757221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77lx\" (UniqueName: \"kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx\") pod \"test-curl-7545-139664773637296\" (UID: \"3f77417d-34f0-438d-aba7-36f84093b3c3\") " pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:53.886795 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:53.886703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:54.017376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:54.017353 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664773637296"] Jul 15 21:47:54.019390 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:47:54.019365 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f77417d_34f0_438d_aba7_36f84093b3c3.slice/crio-9814a17740f079204a4c3acb138a7c1e717b9b9ffffd671c8e07774f5637cc2c WatchSource:0}: Error finding container 9814a17740f079204a4c3acb138a7c1e717b9b9ffffd671c8e07774f5637cc2c: Status 404 returned error can't find the container with id 9814a17740f079204a4c3acb138a7c1e717b9b9ffffd671c8e07774f5637cc2c Jul 15 21:47:54.276688 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:54.276597 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:54.276688 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:54.276622 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f456bed359753a57713f160c087c3d81634fb16904aca612c631b795b2bffe06" Jul 15 21:47:54.277711 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:54.277664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664773637296" event={"ID":"3f77417d-34f0-438d-aba7-36f84093b3c3","Type":"ContainerStarted","Data":"9814a17740f079204a4c3acb138a7c1e717b9b9ffffd671c8e07774f5637cc2c"} Jul 15 21:47:54.287483 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:47:54.287455 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-139664773636656\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:55.284236 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:55.284198 2574 generic.go:356] "Generic (PLEG): container finished" podID="3f77417d-34f0-438d-aba7-36f84093b3c3" containerID="ec35214662ce56f020da5609361a63ff66255a08104780d2aa32e968acec0021" exitCode=0 Jul 15 21:47:55.284622 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:55.284283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664773637296" event={"ID":"3f77417d-34f0-438d-aba7-36f84093b3c3","Type":"ContainerDied","Data":"ec35214662ce56f020da5609361a63ff66255a08104780d2aa32e968acec0021"} Jul 15 21:47:55.292725 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:55.292697 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" path="/var/lib/kubelet/pods/6ec4c847-50d3-4888-90e1-29f30b6bbed6/volumes" Jul 15 21:47:55.305236 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:47:55.305208 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-139664773636656\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="6ec4c847-50d3-4888-90e1-29f30b6bbed6" pod="opendatahub/test-curl-7545-139664773636656" Jul 15 21:47:56.417048 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.417021 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:56.439621 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.439587 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664773637296_3f77417d-34f0-438d-aba7-36f84093b3c3/test-curl-7545-139664773637296/0.log" Jul 15 21:47:56.461044 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.461005 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664773637296"] Jul 15 21:47:56.465467 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.465431 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664773637296"] Jul 15 21:47:56.471516 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.471494 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h77lx\" (UniqueName: \"kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx\") pod \"3f77417d-34f0-438d-aba7-36f84093b3c3\" (UID: \"3f77417d-34f0-438d-aba7-36f84093b3c3\") " Jul 15 21:47:56.473565 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.473537 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx" (OuterVolumeSpecName: "kube-api-access-h77lx") pod "3f77417d-34f0-438d-aba7-36f84093b3c3" (UID: "3f77417d-34f0-438d-aba7-36f84093b3c3"). InnerVolumeSpecName "kube-api-access-h77lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:47:56.573103 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.573011 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77lx\" (UniqueName: \"kubernetes.io/projected/3f77417d-34f0-438d-aba7-36f84093b3c3-kube-api-access-h77lx\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:47:56.618863 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.618824 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:56.619190 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.619175 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f77417d-34f0-438d-aba7-36f84093b3c3" containerName="test-curl-7545-139664773637296" Jul 15 21:47:56.619238 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.619191 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f77417d-34f0-438d-aba7-36f84093b3c3" containerName="test-curl-7545-139664773637296" Jul 15 21:47:56.619273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.619259 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f77417d-34f0-438d-aba7-36f84093b3c3" containerName="test-curl-7545-139664773637296" Jul 15 21:47:56.622109 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.622093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:56.629246 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.629218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:56.673940 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.673895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbjk\" (UniqueName: \"kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk\") pod \"test-curl-7545-139664776715552\" (UID: \"cf61d316-a73c-4e2c-90ab-38fb657980b2\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:56.774814 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.774782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbjk\" (UniqueName: \"kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk\") pod \"test-curl-7545-139664776715552\" (UID: \"cf61d316-a73c-4e2c-90ab-38fb657980b2\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:56.783081 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.783049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbjk\" (UniqueName: \"kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk\") pod \"test-curl-7545-139664776715552\" (UID: \"cf61d316-a73c-4e2c-90ab-38fb657980b2\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:56.822508 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.822473 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:47:56.931377 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:56.931278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:57.060807 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:57.060782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:57.063187 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:47:57.063160 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf61d316_a73c_4e2c_90ab_38fb657980b2.slice/crio-9ba826c30c826d283f4ffb80d8e5957b9c03a099dd483bbd76be6bef49abbefa WatchSource:0}: Error finding container 9ba826c30c826d283f4ffb80d8e5957b9c03a099dd483bbd76be6bef49abbefa: Status 404 returned error can't find the container with id 9ba826c30c826d283f4ffb80d8e5957b9c03a099dd483bbd76be6bef49abbefa Jul 15 21:47:57.292887 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:57.292855 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f77417d-34f0-438d-aba7-36f84093b3c3" path="/var/lib/kubelet/pods/3f77417d-34f0-438d-aba7-36f84093b3c3/volumes" Jul 15 21:47:57.294869 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:57.294838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776715552" event={"ID":"cf61d316-a73c-4e2c-90ab-38fb657980b2","Type":"ContainerStarted","Data":"9ba826c30c826d283f4ffb80d8e5957b9c03a099dd483bbd76be6bef49abbefa"} Jul 15 21:47:57.296004 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:57.295985 2574 scope.go:117] "RemoveContainer" containerID="ec35214662ce56f020da5609361a63ff66255a08104780d2aa32e968acec0021" Jul 15 21:47:57.296102 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:57.296008 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664773637296" Jul 15 21:47:58.303472 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:58.303435 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf61d316-a73c-4e2c-90ab-38fb657980b2" containerID="703ec3294b63abb00d1ef1d7450fbe5e2fff5fb38d8a6ffb4fcb17bad029490f" exitCode=0 Jul 15 21:47:58.303917 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:58.303511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776715552" event={"ID":"cf61d316-a73c-4e2c-90ab-38fb657980b2","Type":"ContainerDied","Data":"703ec3294b63abb00d1ef1d7450fbe5e2fff5fb38d8a6ffb4fcb17bad029490f"} Jul 15 21:47:59.436590 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.436565 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:59.462793 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.462765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664776715552_cf61d316-a73c-4e2c-90ab-38fb657980b2/test-curl-7545-139664776715552/0.log" Jul 15 21:47:59.488891 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.488855 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:59.492312 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.492288 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:59.501808 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.501788 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjbjk\" (UniqueName: \"kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk\") pod \"cf61d316-a73c-4e2c-90ab-38fb657980b2\" (UID: \"cf61d316-a73c-4e2c-90ab-38fb657980b2\") " Jul 15 21:47:59.503843 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.503818 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk" (OuterVolumeSpecName: "kube-api-access-rjbjk") pod "cf61d316-a73c-4e2c-90ab-38fb657980b2" (UID: "cf61d316-a73c-4e2c-90ab-38fb657980b2"). InnerVolumeSpecName "kube-api-access-rjbjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:47:59.603202 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.603116 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbjk\" (UniqueName: \"kubernetes.io/projected/cf61d316-a73c-4e2c-90ab-38fb657980b2-kube-api-access-rjbjk\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:47:59.631913 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.631878 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:59.632219 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.632205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf61d316-a73c-4e2c-90ab-38fb657980b2" containerName="test-curl-7545-139664776715552" Jul 15 21:47:59.632260 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.632221 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf61d316-a73c-4e2c-90ab-38fb657980b2" containerName="test-curl-7545-139664776715552" Jul 15 21:47:59.632296 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.632285 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf61d316-a73c-4e2c-90ab-38fb657980b2" containerName="test-curl-7545-139664776715552" Jul 15 21:47:59.635536 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.635518 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:59.644978 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.644948 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:47:59.704277 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.704241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cnf\" (UniqueName: \"kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf\") pod \"test-curl-7545-139664776715552\" (UID: \"e8593b2b-9035-479c-afef-aca53f98bd04\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:59.805577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.805535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cnf\" (UniqueName: \"kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf\") pod \"test-curl-7545-139664776715552\" (UID: \"e8593b2b-9035-479c-afef-aca53f98bd04\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:59.816204 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.816175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cnf\" (UniqueName: \"kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf\") pod \"test-curl-7545-139664776715552\" (UID: \"e8593b2b-9035-479c-afef-aca53f98bd04\") " pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:47:59.945027 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:47:59.944934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:48:00.082227 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:00.082136 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:48:00.314014 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:00.313976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776715552" event={"ID":"e8593b2b-9035-479c-afef-aca53f98bd04","Type":"ContainerStarted","Data":"cc446558245d4cf4819ccd749643566cb0289e2f40315dfd0cee6aba452ea1ae"} Jul 15 21:48:00.315171 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:00.315149 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba826c30c826d283f4ffb80d8e5957b9c03a099dd483bbd76be6bef49abbefa" Jul 15 21:48:00.315171 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:00.315159 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:48:00.327834 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:00.327799 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7545-139664776715552" oldPodUID="cf61d316-a73c-4e2c-90ab-38fb657980b2" podUID="e8593b2b-9035-479c-afef-aca53f98bd04" Jul 15 21:48:01.293592 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:01.293559 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf61d316-a73c-4e2c-90ab-38fb657980b2" path="/var/lib/kubelet/pods/cf61d316-a73c-4e2c-90ab-38fb657980b2/volumes" Jul 15 21:48:01.321040 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:01.321005 2574 generic.go:356] "Generic (PLEG): container finished" podID="e8593b2b-9035-479c-afef-aca53f98bd04" containerID="d228c1de600237c56d42e374c4c1a1906affa8b68d8c6d83479445bfa0a5d438" exitCode=0 Jul 15 21:48:01.321215 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:01.321071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776715552" event={"ID":"e8593b2b-9035-479c-afef-aca53f98bd04","Type":"ContainerDied","Data":"d228c1de600237c56d42e374c4c1a1906affa8b68d8c6d83479445bfa0a5d438"} Jul 15 21:48:02.457249 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.457225 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:48:02.478557 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.478534 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664776715552_e8593b2b-9035-479c-afef-aca53f98bd04/test-curl-7545-139664776715552/0.log" Jul 15 21:48:02.500753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.500708 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:48:02.502929 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.502903 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664776715552"] Jul 15 21:48:02.531135 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.531105 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2cnf\" (UniqueName: \"kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf\") pod \"e8593b2b-9035-479c-afef-aca53f98bd04\" (UID: \"e8593b2b-9035-479c-afef-aca53f98bd04\") " Jul 15 21:48:02.533175 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.533154 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf" (OuterVolumeSpecName: "kube-api-access-p2cnf") pod "e8593b2b-9035-479c-afef-aca53f98bd04" (UID: "e8593b2b-9035-479c-afef-aca53f98bd04"). InnerVolumeSpecName "kube-api-access-p2cnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:02.621577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.621499 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:02.621874 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.621860 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8593b2b-9035-479c-afef-aca53f98bd04" containerName="test-curl-7545-139664776715552" Jul 15 21:48:02.621934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.621876 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8593b2b-9035-479c-afef-aca53f98bd04" containerName="test-curl-7545-139664776715552" Jul 15 21:48:02.621934 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.621928 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8593b2b-9035-479c-afef-aca53f98bd04" containerName="test-curl-7545-139664776715552" Jul 15 21:48:02.624756 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.624739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:02.631802 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.631461 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:02.631900 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.631874 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cnf\" (UniqueName: \"kubernetes.io/projected/e8593b2b-9035-479c-afef-aca53f98bd04-kube-api-access-p2cnf\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:02.732831 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.732787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzgq\" (UniqueName: \"kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq\") pod \"test-curl-7545-139664776597840\" (UID: \"cd90af1d-1f88-4b56-96e2-4a1a6f744569\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:02.834266 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.834228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzgq\" (UniqueName: \"kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq\") pod \"test-curl-7545-139664776597840\" (UID: \"cd90af1d-1f88-4b56-96e2-4a1a6f744569\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:02.843306 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.843280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzgq\" (UniqueName: \"kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq\") pod \"test-curl-7545-139664776597840\" (UID: \"cd90af1d-1f88-4b56-96e2-4a1a6f744569\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:02.935337 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:02.935246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:03.274657 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.274627 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:03.276563 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:48:03.276535 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd90af1d_1f88_4b56_96e2_4a1a6f744569.slice/crio-7b956149c8ebc307125355eb1a920bc0d110f115f8755160e27ab19b2077e419 WatchSource:0}: Error finding container 7b956149c8ebc307125355eb1a920bc0d110f115f8755160e27ab19b2077e419: Status 404 returned error can't find the container with id 7b956149c8ebc307125355eb1a920bc0d110f115f8755160e27ab19b2077e419 Jul 15 21:48:03.293274 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.293248 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8593b2b-9035-479c-afef-aca53f98bd04" path="/var/lib/kubelet/pods/e8593b2b-9035-479c-afef-aca53f98bd04/volumes" Jul 15 21:48:03.332913 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.332884 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776715552" Jul 15 21:48:03.333091 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.332951 2574 scope.go:117] "RemoveContainer" containerID="d228c1de600237c56d42e374c4c1a1906affa8b68d8c6d83479445bfa0a5d438" Jul 15 21:48:03.334430 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.334335 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776597840" event={"ID":"cd90af1d-1f88-4b56-96e2-4a1a6f744569","Type":"ContainerStarted","Data":"7b956149c8ebc307125355eb1a920bc0d110f115f8755160e27ab19b2077e419"} Jul 15 21:48:03.380325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.380283 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" podUID="e0c9e436-bb85-480c-b0bc-897d75914f0a" containerName="main" containerID="cri-o://5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893" gracePeriod=45 Jul 15 21:48:03.660521 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.660495 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:48:03.742618 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742583 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phphp\" (UniqueName: \"kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.742799 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742629 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.742799 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.742799 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742669 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.742933 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742820 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.742933 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742852 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm\") pod \"e0c9e436-bb85-480c-b0bc-897d75914f0a\" (UID: \"e0c9e436-bb85-480c-b0bc-897d75914f0a\") " Jul 15 21:48:03.743050 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742942 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:03.743050 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.742954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home" (OuterVolumeSpecName: "home") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:03.743158 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.743082 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:03.743236 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.743213 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:03.743280 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.743240 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:03.743280 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.743249 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:03.744762 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.744745 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:48:03.745142 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.745126 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp" (OuterVolumeSpecName: "kube-api-access-phphp") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "kube-api-access-phphp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:03.745200 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.745122 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm" (OuterVolumeSpecName: "dshm") pod "e0c9e436-bb85-480c-b0bc-897d75914f0a" (UID: "e0c9e436-bb85-480c-b0bc-897d75914f0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:03.843928 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.843843 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9e436-bb85-480c-b0bc-897d75914f0a-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:03.843928 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.843873 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phphp\" (UniqueName: \"kubernetes.io/projected/e0c9e436-bb85-480c-b0bc-897d75914f0a-kube-api-access-phphp\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:03.843928 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:03.843883 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9e436-bb85-480c-b0bc-897d75914f0a-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.342148 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.342110 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0c9e436-bb85-480c-b0bc-897d75914f0a" containerID="5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893" exitCode=0 Jul 15 21:48:04.342357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.342159 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" Jul 15 21:48:04.342357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.342201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" event={"ID":"e0c9e436-bb85-480c-b0bc-897d75914f0a","Type":"ContainerDied","Data":"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893"} Jul 15 21:48:04.342357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.342238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv" event={"ID":"e0c9e436-bb85-480c-b0bc-897d75914f0a","Type":"ContainerDied","Data":"9da1d1980446613cb9e0be19fbb3a329e228ce5fd541bbfaa456c15944187fea"} Jul 15 21:48:04.342357 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.342261 2574 scope.go:117] "RemoveContainer" containerID="5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893" Jul 15 21:48:04.343763 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.343726 2574 generic.go:356] "Generic (PLEG): container finished" podID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" containerID="8dd19b54100bc18943f9919c57f77d99ec880f37998c30e67986d9d79a519de2" exitCode=0 Jul 15 21:48:04.343763 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.343764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776597840" event={"ID":"cd90af1d-1f88-4b56-96e2-4a1a6f744569","Type":"ContainerDied","Data":"8dd19b54100bc18943f9919c57f77d99ec880f37998c30e67986d9d79a519de2"} Jul 15 21:48:04.352444 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.352423 2574 scope.go:117] "RemoveContainer" containerID="5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893" Jul 15 21:48:04.352796 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:48:04.352744 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893\": container with ID starting with 5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893 not found: ID does not exist" containerID="5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893" Jul 15 21:48:04.352898 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.352791 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893"} err="failed to get container status \"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893\": rpc error: code = NotFound desc = could not find container \"5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893\": container with ID starting with 5d628458d09b8a94b00fb3ef933f90a94955766e4b6aa1b0c70fa10167587893 not found: ID does not exist" Jul 15 21:48:04.359607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.359579 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" podUID="8750b68c-670a-42f2-b189-51ddbb35b22b" containerName="main" containerID="cri-o://cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d" gracePeriod=45 Jul 15 21:48:04.376006 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.375974 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:48:04.381550 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.381528 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-543174/rate-test-model-543174-kserve-5f6854499f-slxgv"] Jul 15 21:48:04.616132 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.616107 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:48:04.650999 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.650965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651150 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651048 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651150 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651129 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzhgz\" (UniqueName: \"kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651170 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651195 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651267 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache\") pod \"8750b68c-670a-42f2-b189-51ddbb35b22b\" (UID: \"8750b68c-670a-42f2-b189-51ddbb35b22b\") " Jul 15 21:48:04.651420 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home" (OuterVolumeSpecName: "home") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:04.651420 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651346 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:04.651571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:04.651571 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651565 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-home\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.651706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.651582 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-tmp-dir\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.653343 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.653313 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:48:04.653439 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.653339 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz" (OuterVolumeSpecName: "kube-api-access-gzhgz") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "kube-api-access-gzhgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:04.653439 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.653326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm" (OuterVolumeSpecName: "dshm") pod "8750b68c-670a-42f2-b189-51ddbb35b22b" (UID: "8750b68c-670a-42f2-b189-51ddbb35b22b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:04.752632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.752596 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-model-cache\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.752632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.752629 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhgz\" (UniqueName: \"kubernetes.io/projected/8750b68c-670a-42f2-b189-51ddbb35b22b-kube-api-access-gzhgz\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.752632 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.752640 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8750b68c-670a-42f2-b189-51ddbb35b22b-dshm\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:04.753079 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:04.752650 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8750b68c-670a-42f2-b189-51ddbb35b22b-tls-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:05.292953 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.292917 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c9e436-bb85-480c-b0bc-897d75914f0a" path="/var/lib/kubelet/pods/e0c9e436-bb85-480c-b0bc-897d75914f0a/volumes" Jul 15 21:48:05.349756 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.349719 2574 generic.go:356] "Generic (PLEG): container finished" podID="8750b68c-670a-42f2-b189-51ddbb35b22b" containerID="cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d" exitCode=0 Jul 15 21:48:05.349964 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.349768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" Jul 15 21:48:05.349964 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.349807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" event={"ID":"8750b68c-670a-42f2-b189-51ddbb35b22b","Type":"ContainerDied","Data":"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d"} Jul 15 21:48:05.349964 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.349862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8" event={"ID":"8750b68c-670a-42f2-b189-51ddbb35b22b","Type":"ContainerDied","Data":"acae3e46f5f7aab596d350371d23ed84487714841da41ffa2dc754c0e2685caa"} Jul 15 21:48:05.349964 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.349880 2574 scope.go:117] "RemoveContainer" containerID="cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d" Jul 15 21:48:05.359317 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.359295 2574 scope.go:117] "RemoveContainer" containerID="cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d" Jul 15 21:48:05.359709 ip-10-0-139-183 kubenswrapper[2574]: E0715 21:48:05.359639 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d\": container with ID starting with cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d not found: ID does not exist" containerID="cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d" Jul 15 21:48:05.359815 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.359710 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d"} err="failed to get container status \"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d\": rpc error: code = NotFound desc = could not find container \"cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d\": container with ID starting with cf73fc1e709fd98ef44df1ddb7ab512ee71867191125099e2517dfe82e59b55d not found: ID does not exist" Jul 15 21:48:05.373577 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.373541 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:48:05.378080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.378039 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-915a1c/rate-test-model-915a1c-kserve-86c44b4ddc-v2cw8"] Jul 15 21:48:05.489169 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.489143 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:05.513936 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.513906 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664776597840_cd90af1d-1f88-4b56-96e2-4a1a6f744569/test-curl-7545-139664776597840/0.log" Jul 15 21:48:05.539018 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.538977 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:05.542690 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.542652 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:05.559233 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.559167 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fzgq\" (UniqueName: \"kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq\") pod \"cd90af1d-1f88-4b56-96e2-4a1a6f744569\" (UID: \"cd90af1d-1f88-4b56-96e2-4a1a6f744569\") " Jul 15 21:48:05.561195 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.561170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq" (OuterVolumeSpecName: "kube-api-access-6fzgq") pod "cd90af1d-1f88-4b56-96e2-4a1a6f744569" (UID: "cd90af1d-1f88-4b56-96e2-4a1a6f744569"). InnerVolumeSpecName "kube-api-access-6fzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:05.660063 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.660022 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fzgq\" (UniqueName: \"kubernetes.io/projected/cd90af1d-1f88-4b56-96e2-4a1a6f744569-kube-api-access-6fzgq\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:05.672787 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.672750 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:05.673101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673089 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" containerName="test-curl-7545-139664776597840" Jul 15 21:48:05.673156 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673102 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" containerName="test-curl-7545-139664776597840" Jul 15 21:48:05.673156 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673115 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8750b68c-670a-42f2-b189-51ddbb35b22b" containerName="main" Jul 15 21:48:05.673156 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673120 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8750b68c-670a-42f2-b189-51ddbb35b22b" containerName="main" Jul 15 21:48:05.673156 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673127 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c9e436-bb85-480c-b0bc-897d75914f0a" containerName="main" Jul 15 21:48:05.673156 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673132 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c9e436-bb85-480c-b0bc-897d75914f0a" containerName="main" Jul 15 21:48:05.673316 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673184 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c9e436-bb85-480c-b0bc-897d75914f0a" containerName="main" Jul 15 21:48:05.673316 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673194 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8750b68c-670a-42f2-b189-51ddbb35b22b" containerName="main" Jul 15 21:48:05.673316 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.673201 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" containerName="test-curl-7545-139664776597840" Jul 15 21:48:05.676066 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.676050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:05.684113 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.684083 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:05.760407 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.760372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bbv\" (UniqueName: \"kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv\") pod \"test-curl-7545-139664776597840\" (UID: \"cc219e18-d4d6-4ae5-90f6-5874b73d59c7\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:05.861713 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.861601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bbv\" (UniqueName: \"kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv\") pod \"test-curl-7545-139664776597840\" (UID: \"cc219e18-d4d6-4ae5-90f6-5874b73d59c7\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:05.871160 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.871136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bbv\" (UniqueName: \"kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv\") pod \"test-curl-7545-139664776597840\" (UID: \"cc219e18-d4d6-4ae5-90f6-5874b73d59c7\") " pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:05.985858 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:05.985818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:06.117619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:06.117541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:06.120450 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:48:06.120420 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc219e18_d4d6_4ae5_90f6_5874b73d59c7.slice/crio-0220c7630a189faa929ab2e7b3f90fd844b23c9370922f87a65b0f363b24da20 WatchSource:0}: Error finding container 0220c7630a189faa929ab2e7b3f90fd844b23c9370922f87a65b0f363b24da20: Status 404 returned error can't find the container with id 0220c7630a189faa929ab2e7b3f90fd844b23c9370922f87a65b0f363b24da20 Jul 15 21:48:06.357221 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:06.357196 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:06.357438 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:06.357242 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b956149c8ebc307125355eb1a920bc0d110f115f8755160e27ab19b2077e419" Jul 15 21:48:06.358466 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:06.358443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776597840" event={"ID":"cc219e18-d4d6-4ae5-90f6-5874b73d59c7","Type":"ContainerStarted","Data":"0220c7630a189faa929ab2e7b3f90fd844b23c9370922f87a65b0f363b24da20"} Jul 15 21:48:06.369265 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:06.369206 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7545-139664776597840" oldPodUID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" podUID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" Jul 15 21:48:07.293308 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:07.293272 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8750b68c-670a-42f2-b189-51ddbb35b22b" path="/var/lib/kubelet/pods/8750b68c-670a-42f2-b189-51ddbb35b22b/volumes" Jul 15 21:48:07.293738 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:07.293590 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd90af1d-1f88-4b56-96e2-4a1a6f744569" path="/var/lib/kubelet/pods/cd90af1d-1f88-4b56-96e2-4a1a6f744569/volumes" Jul 15 21:48:07.364043 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:07.364012 2574 generic.go:356] "Generic (PLEG): container finished" podID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" containerID="cd2b42e53f0a040b585e37d52fa92764b3043bc61a43ca5dc636753963353de1" exitCode=0 Jul 15 21:48:07.364187 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:07.364058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-139664776597840" event={"ID":"cc219e18-d4d6-4ae5-90f6-5874b73d59c7","Type":"ContainerDied","Data":"cd2b42e53f0a040b585e37d52fa92764b3043bc61a43ca5dc636753963353de1"} Jul 15 21:48:08.509749 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.508597 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:08.534082 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.534050 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7545-139664776597840_cc219e18-d4d6-4ae5-90f6-5874b73d59c7/test-curl-7545-139664776597840/0.log" Jul 15 21:48:08.554629 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.554589 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:08.558642 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.558612 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-139664776597840"] Jul 15 21:48:08.584273 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.584248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7bbv\" (UniqueName: \"kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv\") pod \"cc219e18-d4d6-4ae5-90f6-5874b73d59c7\" (UID: \"cc219e18-d4d6-4ae5-90f6-5874b73d59c7\") " Jul 15 21:48:08.586280 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.586252 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv" (OuterVolumeSpecName: "kube-api-access-p7bbv") pod "cc219e18-d4d6-4ae5-90f6-5874b73d59c7" (UID: "cc219e18-d4d6-4ae5-90f6-5874b73d59c7"). InnerVolumeSpecName "kube-api-access-p7bbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:08.685199 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:08.685121 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7bbv\" (UniqueName: \"kubernetes.io/projected/cc219e18-d4d6-4ae5-90f6-5874b73d59c7-kube-api-access-p7bbv\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 15 21:48:09.111626 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:09.111592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:09.293174 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:09.293131 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" path="/var/lib/kubelet/pods/cc219e18-d4d6-4ae5-90f6-5874b73d59c7/volumes" Jul 15 21:48:09.375996 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:09.375912 2574 scope.go:117] "RemoveContainer" containerID="cd2b42e53f0a040b585e37d52fa92764b3043bc61a43ca5dc636753963353de1" Jul 15 21:48:09.376142 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:09.375918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-139664776597840" Jul 15 21:48:15.157397 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:15.157346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:15.257616 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:15.257562 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:15.462999 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:15.462619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:15.684650 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:15.684609 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:16.155354 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:16.155319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:19.415385 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:19.415350 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:48:22.055841 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:22.055808 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:22.117064 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:22.117034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:22.192288 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:22.192248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:22.429443 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:22.429355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:22.838272 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:22.838233 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76f848cf5-qdk6j_14965a65-b377-4d8d-9a6e-98b4a44a8a78/authorino/0.log" Jul 15 21:48:29.054818 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:29.054791 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f656749fc-lk5zk_396ac956-aeb0-4d62-80e6-94c59aa22fbe/copy-manifests/0.log" Jul 15 21:48:29.063925 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:29.063886 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f656749fc-lk5zk_396ac956-aeb0-4d62-80e6-94c59aa22fbe/manager/0.log" Jul 15 21:48:30.968503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:30.968466 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735865-smkj9_b37009bb-abf1-4c26-8e48-89813a87fb37/cleanup/0.log" Jul 15 21:48:31.115422 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:31.115383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ghvfw"] Jul 15 21:48:31.527475 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:31.527443 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-da1123-29735865-6ppm8_a8bfa61b-6cf2-4c67-b5ca-69645e7af648/cleanup/0.log" Jul 15 21:48:33.527181 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:33.527151 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76f848cf5-qdk6j_14965a65-b377-4d8d-9a6e-98b4a44a8a78/authorino/0.log" Jul 15 21:48:33.758975 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:33.758946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-99pwr_59271228-bf3a-4908-94de-2c05b8c05bf3/manager/0.log" Jul 15 21:48:34.235979 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:34.235953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ghvfw_2adf7ba2-7cd0-4811-b78e-7911c879f485/limitador/0.log" Jul 15 21:48:34.365493 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:34.365461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4tzss_6926776c-3988-47cd-b382-d847cbb41a9a/manager/0.log" Jul 15 21:48:35.105384 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:35.105339 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fdd66594-v6rjh_7484e6b7-9867-4ab0-9569-97ce4c12597a/kube-auth-proxy/0.log" Jul 15 21:48:35.375756 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:35.375596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:35.475422 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:35.475389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:36.398739 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:36.398705 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-kncdr_a677a8dd-bb7b-402b-bb6f-59c1e444866f/main/0.log" Jul 15 21:48:36.511799 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:36.511770 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-hx7z5_1cdce663-a6c9-4a7d-904d-e283d4657127/main/0.log" Jul 15 21:48:42.204234 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.204189 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2496/must-gather-26zlv"] Jul 15 21:48:42.204607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.204553 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" containerName="test-curl-7545-139664776597840" Jul 15 21:48:42.204607 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.204564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" containerName="test-curl-7545-139664776597840" Jul 15 21:48:42.204706 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.204621 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc219e18-d4d6-4ae5-90f6-5874b73d59c7" containerName="test-curl-7545-139664776597840" Jul 15 21:48:42.207616 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.207593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.210496 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.210476 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v2496\"/\"default-dockercfg-czhdx\"" Jul 15 21:48:42.210609 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.210588 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2496\"/\"kube-root-ca.crt\"" Jul 15 21:48:42.210718 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.210700 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2496\"/\"openshift-service-ca.crt\"" Jul 15 21:48:42.215764 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.215745 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2496/must-gather-26zlv"] Jul 15 21:48:42.218760 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.218711 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:42.278101 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.278070 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:42.286398 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.286375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bsw\" (UniqueName: \"kubernetes.io/projected/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-kube-api-access-44bsw\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.286468 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.286403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-must-gather-output\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.387671 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.387636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bsw\" (UniqueName: \"kubernetes.io/projected/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-kube-api-access-44bsw\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.387836 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.387698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-must-gather-output\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.388056 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.388038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-must-gather-output\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.396591 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.396564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bsw\" (UniqueName: \"kubernetes.io/projected/b93eaeb4-6eae-4933-8a00-f3324ddc84b0-kube-api-access-44bsw\") pod \"must-gather-26zlv\" (UID: \"b93eaeb4-6eae-4933-8a00-f3324ddc84b0\") " pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.517529 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.517420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2496/must-gather-26zlv" Jul 15 21:48:42.699587 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:42.699555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2496/must-gather-26zlv"] Jul 15 21:48:42.703898 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:48:42.703871 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93eaeb4_6eae_4933_8a00_f3324ddc84b0.slice/crio-99c288eda51363a6ce9363c3f65bea0c9024467405f8537f14876f759d93a61a WatchSource:0}: Error finding container 99c288eda51363a6ce9363c3f65bea0c9024467405f8537f14876f759d93a61a: Status 404 returned error can't find the container with id 99c288eda51363a6ce9363c3f65bea0c9024467405f8537f14876f759d93a61a Jul 15 21:48:43.537948 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:43.537905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2496/must-gather-26zlv" event={"ID":"b93eaeb4-6eae-4933-8a00-f3324ddc84b0","Type":"ContainerStarted","Data":"99c288eda51363a6ce9363c3f65bea0c9024467405f8537f14876f759d93a61a"} Jul 15 21:48:44.544436 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:44.544400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2496/must-gather-26zlv" event={"ID":"b93eaeb4-6eae-4933-8a00-f3324ddc84b0","Type":"ContainerStarted","Data":"fd2a121e99ee4738dced8ad191cf958bb9f7e890e87acd8a3c7c9bb8051a7727"} Jul 15 21:48:44.544897 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:44.544449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2496/must-gather-26zlv" event={"ID":"b93eaeb4-6eae-4933-8a00-f3324ddc84b0","Type":"ContainerStarted","Data":"980bb3549ec9d21f01be44f6968bb2488ca62510249e403e5247bd3d91264287"} Jul 15 21:48:44.566410 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:44.566344 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v2496/must-gather-26zlv" podStartSLOduration=1.6345330219999998 podStartE2EDuration="2.566323603s" totalImagesPullingTime="931.790581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:48:44.56261832 +0000 UTC m=+2319.803764810" watchObservedRunningTime="2026-07-15 21:48:44.566323603 +0000 UTC m=+2319.807470092" Jul 15 21:48:45.256992 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:45.256963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vbfsf_64381618-e5ba-4a2d-aeeb-c435bffe9ead/global-pull-secret-syncer/0.log" Jul 15 21:48:45.306430 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:45.306398 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8kgdp_dcb762e6-ed5a-4972-b65e-97ac73fad4fa/konnectivity-agent/0.log" Jul 15 21:48:45.432844 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:45.432809 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-183.ec2.internal_444dd7fe6c8562d1e4c53584c89f8ea5/haproxy/0.log" Jul 15 21:48:50.793549 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:50.793512 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76f848cf5-qdk6j_14965a65-b377-4d8d-9a6e-98b4a44a8a78/authorino/0.log" Jul 15 21:48:50.855864 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:50.855837 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-99pwr_59271228-bf3a-4908-94de-2c05b8c05bf3/manager/0.log" Jul 15 21:48:51.044655 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:51.044572 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ghvfw_2adf7ba2-7cd0-4811-b78e-7911c879f485/limitador/0.log" Jul 15 21:48:51.118924 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:51.118892 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4tzss_6926776c-3988-47cd-b382-d847cbb41a9a/manager/0.log" Jul 15 21:48:52.873350 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:52.873269 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/alertmanager/0.log" Jul 15 21:48:52.912568 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:52.912542 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/config-reloader/0.log" Jul 15 21:48:52.942528 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:52.942496 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/kube-rbac-proxy-web/0.log" Jul 15 21:48:52.967831 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:52.967797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/kube-rbac-proxy/0.log" Jul 15 21:48:52.994246 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:52.994207 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/kube-rbac-proxy-metric/0.log" Jul 15 21:48:53.018080 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.018060 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/prom-label-proxy/0.log" Jul 15 21:48:53.043455 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.043429 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_08d88ef0-1ff1-4dfc-a820-995124843411/init-config-reloader/0.log" Jul 15 21:48:53.194075 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.194026 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57746d6954-p74xg_0109c5b5-ccb1-45ed-9f8d-f0b33e00f41c/metrics-server/0.log" Jul 15 21:48:53.261376 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.261346 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rsxl_4354908d-9298-47c3-8ed2-d15b99b2f47f/node-exporter/0.log" Jul 15 21:48:53.283469 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.283422 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rsxl_4354908d-9298-47c3-8ed2-d15b99b2f47f/kube-rbac-proxy/0.log" Jul 15 21:48:53.307057 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.307029 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rsxl_4354908d-9298-47c3-8ed2-d15b99b2f47f/init-textfile/0.log" Jul 15 21:48:53.583718 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.583667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/prometheus/0.log" Jul 15 21:48:53.608981 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.608955 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/config-reloader/0.log" Jul 15 21:48:53.631727 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.631667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/thanos-sidecar/0.log" Jul 15 21:48:53.656325 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.656289 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/kube-rbac-proxy-web/0.log" Jul 15 21:48:53.678940 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.678908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/kube-rbac-proxy/0.log" Jul 15 21:48:53.700503 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.700475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/kube-rbac-proxy-thanos/0.log" Jul 15 21:48:53.725300 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.725274 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d009018d-540d-4dbd-9eca-8df5283d234c/init-config-reloader/0.log" Jul 15 21:48:53.753686 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.753642 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-wb8bh_b3d783de-ce9b-4904-ae2b-23d5746e2ed9/prometheus-operator/0.log" Jul 15 21:48:53.770604 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.770573 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-wb8bh_b3d783de-ce9b-4904-ae2b-23d5746e2ed9/kube-rbac-proxy/0.log" Jul 15 21:48:53.798831 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:53.798801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-pcq8d_ac2244ee-2eb4-48fe-8fa5-20cf5bc0a2be/prometheus-operator-admission-webhook/0.log" Jul 15 21:48:55.298082 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.298050 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw"] Jul 15 21:48:55.306569 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.306335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.307854 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.307825 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw"] Jul 15 21:48:55.411984 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.411938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-sys\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.412173 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.412028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzct\" (UniqueName: \"kubernetes.io/projected/580b4452-f621-41ac-bac6-ac46377a4f21-kube-api-access-mlzct\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.412173 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.412150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-proc\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.412304 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.412251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-lib-modules\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.412369 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.412325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-podres\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513427 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-sys\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzct\" (UniqueName: \"kubernetes.io/projected/580b4452-f621-41ac-bac6-ac46377a4f21-kube-api-access-mlzct\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-proc\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-lib-modules\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-sys\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513619 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-podres\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513599 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-proc\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-lib-modules\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.513868 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.513709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/580b4452-f621-41ac-bac6-ac46377a4f21-podres\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.522183 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.522154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzct\" (UniqueName: \"kubernetes.io/projected/580b4452-f621-41ac-bac6-ac46377a4f21-kube-api-access-mlzct\") pod \"perf-node-gather-daemonset-wrtzw\" (UID: \"580b4452-f621-41ac-bac6-ac46377a4f21\") " pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.621705 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.621594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:55.823225 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:55.823182 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw"] Jul 15 21:48:55.824765 ip-10-0-139-183 kubenswrapper[2574]: W0715 21:48:55.824735 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod580b4452_f621_41ac_bac6_ac46377a4f21.slice/crio-31387eed44cf8b82baae143482e2e7d06d34023cbdfde81af5cb0db2518f4740 WatchSource:0}: Error finding container 31387eed44cf8b82baae143482e2e7d06d34023cbdfde81af5cb0db2518f4740: Status 404 returned error can't find the container with id 31387eed44cf8b82baae143482e2e7d06d34023cbdfde81af5cb0db2518f4740 Jul 15 21:48:56.630119 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:56.630082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" event={"ID":"580b4452-f621-41ac-bac6-ac46377a4f21","Type":"ContainerStarted","Data":"dddc95238eadab49489b020c6d6a97928cb8b1f1142690a818ea3f6e5aadc082"} Jul 15 21:48:56.630589 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:56.630132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" event={"ID":"580b4452-f621-41ac-bac6-ac46377a4f21","Type":"ContainerStarted","Data":"31387eed44cf8b82baae143482e2e7d06d34023cbdfde81af5cb0db2518f4740"} Jul 15 21:48:56.630589 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:56.630212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:48:56.649401 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:56.649294 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" podStartSLOduration=1.649271315 podStartE2EDuration="1.649271315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:48:56.648562167 +0000 UTC m=+2331.889708651" watchObservedRunningTime="2026-07-15 21:48:56.649271315 +0000 UTC m=+2331.890417806" Jul 15 21:48:58.009753 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:58.009724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sw96t_8847d8c3-454a-464e-9eac-52ed4d29a323/dns/0.log" Jul 15 21:48:58.030403 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:58.030378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sw96t_8847d8c3-454a-464e-9eac-52ed4d29a323/kube-rbac-proxy/0.log" Jul 15 21:48:58.100991 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:58.100957 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tb2fz_9469fb72-dc7b-438a-8d69-5c0fc6aff7da/dns-node-resolver/0.log" Jul 15 21:48:58.672602 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:58.672562 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s67rl_ce512db5-dc5d-43ff-acd1-5e7f20b33068/node-ca/0.log" Jul 15 21:48:59.771788 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:48:59.771761 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fdd66594-v6rjh_7484e6b7-9867-4ab0-9569-97ce4c12597a/kube-auth-proxy/0.log" Jul 15 21:49:00.632947 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:00.632919 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shs46_1f193cd6-0f42-4248-9524-1561e6e59caa/serve-healthcheck-canary/0.log" Jul 15 21:49:01.177804 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:01.177774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lww7q_afef0e38-49c4-4f15-ba38-b922166982d4/kube-rbac-proxy/0.log" Jul 15 21:49:01.197810 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:01.197787 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lww7q_afef0e38-49c4-4f15-ba38-b922166982d4/exporter/0.log" Jul 15 21:49:01.218541 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:01.218516 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lww7q_afef0e38-49c4-4f15-ba38-b922166982d4/extractor/0.log" Jul 15 21:49:02.646582 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:02.646552 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v2496/perf-node-gather-daemonset-wrtzw" Jul 15 21:49:03.770389 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:03.770361 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f656749fc-lk5zk_396ac956-aeb0-4d62-80e6-94c59aa22fbe/manager/0.log" Jul 15 21:49:03.788784 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:03.788760 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f656749fc-lk5zk_396ac956-aeb0-4d62-80e6-94c59aa22fbe/copy-manifests/0.log" Jul 15 21:49:05.381550 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:05.381517 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-764bf7f9d4-qh6ff_2f2cdbfb-badc-4465-ac19-fa7fa73b2718/manager/0.log" Jul 15 21:49:05.901694 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:05.901654 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fdd66594-v6rjh_7484e6b7-9867-4ab0-9569-97ce4c12597a/kube-auth-proxy/0.log" Jul 15 21:49:07.249029 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:07.248945 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fdd66594-v6rjh_7484e6b7-9867-4ab0-9569-97ce4c12597a/kube-auth-proxy/0.log" Jul 15 21:49:11.541543 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.541488 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/kube-multus-additional-cni-plugins/0.log" Jul 15 21:49:11.563107 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.563085 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/egress-router-binary-copy/0.log" Jul 15 21:49:11.582096 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.582076 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/cni-plugins/0.log" Jul 15 21:49:11.603425 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.603408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/bond-cni-plugin/0.log" Jul 15 21:49:11.625774 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.625752 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/routeoverride-cni/0.log" Jul 15 21:49:11.647582 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.647558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/whereabouts-cni-bincopy/0.log" Jul 15 21:49:11.667129 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:11.667106 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p2q6_fc2f32a9-f96f-4c0b-9480-1d54dc1e7b87/whereabouts-cni/0.log" Jul 15 21:49:12.028601 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:12.028520 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k4m4z_c8701063-1150-4ed0-b93d-df0fec133df0/kube-multus/0.log" Jul 15 21:49:12.082360 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:12.082337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6d8qf_2f49ab87-f4af-4a57-a28c-abb49fd76ca4/network-metrics-daemon/0.log" Jul 15 21:49:12.100150 ip-10-0-139-183 kubenswrapper[2574]: I0715 21:49:12.100119 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6d8qf_2f49ab87-f4af-4a57-a28c-abb49fd76ca4/kube-rbac-proxy/0.log"